Predicting and responding to cut in vehicles and altruistic responses

ABSTRACT

A vehicle navigation system may comprise a memory including instructions and circuitry configured by the instructions to identify a target vehicle in an environment of a vehicle that includes the vehicle navigation system. The circuitry may receive image data of the target vehicle from at least one image capture device of the vehicle; identify, based on analysis of the image data, one or more situational characteristics of the target vehicle, the situational characteristics of the target including an indication that the target vehicle has an activated blinker; and change a navigational state of the vehicle to allow an action of the target vehicle. The vehicle may be configured to cause the change in the navigational state based on a determination that the one or more situational characteristics, including the activated blinker, indicate that the target vehicle would benefit from the change in the navigational state.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalPatent Application No. 62/260,281, filed on Nov. 26, 2015, and U.S.Provisional Patent Application No. 62/361,343, filed on Jul. 12, 2016.All of the foregoing applications are incorporated herein by referencein their entirety.

BACKGROUND Technical Field

The present disclosure relates generally to autonomous vehiclenavigation.

Additionally, this disclosure relates to systems and methods fordetecting and responding to cut in vehicles, and navigating while takinginto consideration an altruistic behavior parameter.

Background Information

As technology continues to advance, the goal of a fully autonomousvehicle that is capable of navigating on roadways is on the horizon.Autonomous vehicles may need to take into account a variety of factorsand make appropriate decisions based on those factors to safely andaccurately reach an intended destination. For example, an autonomousvehicle may need to process and interpret visual information (e.g.,information captured from a camera) and may also use informationobtained from other sources (e.g., from a global positioning system(GPS) device, a speed sensor, an accelerometer, a suspension sensor,etc.). At the same time, in order to navigate to a destination, anautonomous vehicle may also need to identify its location within aparticular roadway (e.g., a specific lane within a multi-lane road),navigate alongside other vehicles, avoid obstacles and pedestrians,observe traffic signals and signs, and travel from one road to anotherroad at appropriate intersections or interchanges.

During navigation, an autonomous vehicle may encounter another vehiclethat is attempting a lane shift. For example, a vehicle in a lane to theleft or to the right of the lane in which the autonomous vehicle istraveling may attempt to shift, or cut in, to the lane in which theautonomous vehicle is traveling. When such a cut in occurs, theautonomous vehicle must make a navigational response by, for example,changing its velocity or acceleration and/or shifting to another lane toavoid the cut-in by the other vehicle.

In some instances, the other vehicle may appear to attempt a cut in, butthe cut in may ultimately not be completed (e.g., because a driver ofthe other vehicle changes his or her mind or the other vehicle is simplydrifting). While delaying effecting a navigational response until a cutin by the other vehicle is sufficiently likely to occur may preventunnecessary braking, such a delay may also increase the risk of acollision and/or result in braking that may cause discomfort to a personin the autonomous vehicle. Thus, improved prediction of when a vehiclewill attempt a cut in is needed.

Moreover, in some cases, a cut in by the other vehicle may benecessitated, e.g., by the roadway and/or traffic rules. In other cases,though, the cut in may be optional, such as when the other vehiclemerely wishes to pass a slower moving vehicle. Because an autonomousvehicle may be programmed to travel to a destination in a timely andsafe manner, the autonomous vehicle may not necessarily permit the othervehicle to cut in where the cut in is not necessary. In some cases,however, it may be preferable to an operator of the autonomous vehicleand/or for overall traffic efficiency to allow such a cut in. Thus, acut in process that encompasses altruistic behavior is needed.

SUMMARY

Embodiments consistent with the present disclosure provide systems andmethods for autonomous vehicle navigation. The disclosed embodiments mayuse cameras to provide autonomous vehicle navigation features. Forexample, consistent with the disclosed embodiments, the disclosedsystems may include one, two, or more cameras that monitor theenvironment of a vehicle. The disclosed systems may provide anavigational response based on, for example, an analysis of imagescaptured by one or more of the cameras. The navigational response mayalso take into account other data including, for example, globalpositioning system (GPS) data, sensor data (e.g., from an accelerometer,a speed sensor, a suspension sensor, etc.), and/or other map data.

Consistent with a disclosed embodiment, a vehicle cut in detection andresponse system for a host vehicle is provided. The system may include adata interface and at least one processing device. The at least oneprocessing device may be programmed to receive, via the data interface,a plurality of images from at least one image capture device associatedwith the host vehicle; identify, in the plurality of images, arepresentation of a target vehicle traveling in a first lane differentfrom a second lane in which the host vehicle is traveling; identify,based on analysis of the plurality of images, at least one indicatorthat the target vehicle will change from the first lane to the secondlane; detect whether at least one predetermined cut in sensitivitychange factor is present in an environment of the host vehicle; cause afirst navigational response in the host vehicle based on theidentification of the at least one indicator and based on a valueassociated with a first cut in sensitivity parameter where nopredetermined cut in sensitivity change factor is detected; and cause asecond navigational response in the host vehicle based on theidentification of the at least one indicator and based on a valueassociated with a second cut in sensitivity parameter where the at leastone predetermined cut in sensitivity change factor is detected, thesecond cut in sensitivity parameter being different from the first cutin sensitivity parameter.

Consistent with another disclosed embodiment, a host vehicle may includea body, at least one image capture device, and at least one processingdevice. The at least one processing device may be programmed to receivea plurality of images from the at least one image capture device;identify, in the plurality of images, a representation of a targetvehicle traveling in a first lane different from a second lane in whichthe host vehicle is traveling; identify, based on analysis of theplurality of images, at least one indicator that the target vehicle willchange from the first lane to the second lane; detect whether at leastone predetermined cut in sensitivity change factor is present in anenvironment of the host vehicle; cause a first navigational response inthe host vehicle based on the identification of the at least oneindicator and based on a value associated with a first cut insensitivity parameter where no predetermined cut in sensitivity changefactor is detected; and cause a second navigational response in the hostvehicle based on the identification of the at least one indicator andbased on a value associated with a second cut in sensitivity parameterwhere the at least one predetermined cut in sensitivity change factor isdetected, the second cut in sensitivity parameter being different fromthe first cut in sensitivity parameter.

Consistent with yet another disclosed embodiment, a method is providedfor detecting and responding to a cut in by a target vehicle. The methodmay include receiving, a plurality of images from at least one imagecapture device associated with a host vehicle;

identifying, in the plurality of images, a representation of the targetvehicle traveling in a first lane different from a second lane in whichthe host vehicle is traveling; identifying, based on analysis of theplurality of images, at least one indicator that the target vehicle willchange from the first lane to the second lane; detecting whether atleast one predetermined cut in sensitivity change factor is present inan environment of the host vehicle; causing a first navigationalresponse in the host vehicle based on the identification of the at leastone indicator and based on a value associated with a first cut insensitivity parameter where no predetermined cut in sensitivity changefactor is detected; and causing a second navigational response in thehost vehicle based on the identification of the at least one indicatorand based on a value associated with a second cut in sensitivityparameter where the at least one predetermined cut in sensitivity changefactor is detected, the second cut in sensitivity parameter beingdifferent from the first cut in sensitivity parameter.

Consistent with a disclosed embodiment, a navigation system is providedfor a host vehicle. The system may include a data interface and at leastone processing device. The at least one processing device may beprogrammed to receive, via the data interface, a plurality of imagesfrom at least one image capture device associated with the host vehicle;identify, based on analysis of the plurality of images, at least onetarget vehicle in an environment of the host vehicle; determine, basedon analysis of the plurality of images, one or more situationalcharacteristics associated with the target vehicle; determine a currentvalue associated with an altruistic behavior parameter; and determinebased on the one or more situational characteristics associated with thetarget vehicle that no change in a navigation state of the host vehicleis required, but cause at least one navigational change in the hostvehicle based on the current value associated with the altruisticbehavior parameter and based on the one or more situationalcharacteristics associated with the target vehicle.

Consistent with another disclosed embodiment, a host vehicle may includea body, at least one image capture device, and at least one processingdevice. The at least one processing device may be configured to receivea plurality of images from the at least one image capture device;identify, based on analysis of the plurality of images, at least onetarget vehicle in an environment of the host vehicle; determine, basedon analysis of the plurality of images, one or more situationalcharacteristics associated with the target vehicle; determine a currentvalue associated with an altruistic behavior parameter; and determinebased on the one or more situational characteristics associated with thetarget vehicle that no change in a navigation state of the host vehicleis required, but cause at least one navigational change in the hostvehicle based on the current value associated with the altruisticbehavior parameter and based on the one or more situationalcharacteristics associated with the target vehicle.

Consistent with yet another disclosed embodiment, a method is providedfor navigating a host vehicle. The method may include receiving aplurality of images from at least one image capture device associatedwith the vehicle; identifying, based on analysis of the plurality ofimages, at least one target vehicle in an environment of the hostvehicle; determining, based on analysis of the plurality of images, oneor more situational characteristics associated with the target vehicle;determining a current value associated with an altruistic behaviorparameter; and determining based on the one or more situationalcharacteristics associated with the target vehicle that no change in anavigation state of the host vehicle is required, but cause at least onenavigational change in the host vehicle based on the current valueassociated with the altruistic behavior parameter and based on the oneor more situational characteristics associated with the target vehicle.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store program instructions, whichare executed by at least one processing device and perform any of themethods described herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1 is a diagrammatic representation of an exemplary systemconsistent with the disclosed embodiments.

FIG. 2A is a diagrammatic side view representation of an exemplaryvehicle including a system consistent with the disclosed embodiments.

FIG. 2B is a diagrammatic top view representation of the vehicle andsystem shown in FIG. 2A consistent with the disclosed embodiments.

FIG. 2C is a diagrammatic top view representation of another embodimentof a vehicle including a system consistent with the disclosedembodiments.

FIG. 2D is a diagrammatic top view representation of yet anotherembodiment of a vehicle including a system consistent with the disclosedembodiments.

FIG. 2E is a diagrammatic top view representation of yet anotherembodiment of a vehicle including a system consistent with the disclosedembodiments.

FIG. 2F is a diagrammatic representation of exemplary vehicle controlsystems consistent with the disclosed embodiments.

FIG. 3A is a diagrammatic representation of an interior of a vehicleincluding a rearview mirror and a user interface for a vehicle imagingsystem consistent with the disclosed embodiments.

FIG. 3B is an illustration of an example of a camera mount that isconfigured to be positioned behind a rearview mirror and against avehicle windshield consistent with the disclosed embodiments.

FIG. 3C is an illustration of the camera mount shown in FIG. 3B from adifferent perspective consistent with the disclosed embodiments.

FIG. 3D is an illustration of an example of a camera mount that isconfigured to be positioned behind a rearview mirror and against avehicle windshield consistent with the disclosed embodiments.

FIG. 4 is an exemplary block diagram of a memory configured to storeinstructions for performing one or more operations consistent with thedisclosed embodiments.

FIG. 5A is a flowchart showing an exemplary process for causing one ormore navigational responses based on monocular image analysis consistentwith disclosed embodiments.

FIG. 5B is a flowchart showing an exemplary process for detecting one ormore vehicles and/or pedestrians in a set of images consistent with thedisclosed embodiments.

FIG. 5C is a flowchart showing an exemplary process for detecting roadmarks and/or lane geometry information in a set of images consistentwith the disclosed embodiments.

FIG. 5D is a flowchart showing an exemplary process for detectingtraffic lights in a set of images consistent with the disclosedembodiments.

FIG. 5E is a flowchart showing an exemplary process for causing one ormore navigational responses based on a vehicle path consistent with thedisclosed embodiments.

FIG. 5F is a flowchart showing an exemplary process for determiningwhether a leading vehicle is changing lanes consistent with thedisclosed embodiments.

FIG. 6 is a flowchart showing an exemplary process for causing one ormore navigational responses based on stereo image analysis consistentwith the disclosed embodiments.

FIG. 7 is a flowchart showing an exemplary process for causing one ormore navigational responses based on an analysis of three sets of imagesconsistent with the disclosed embodiments.

FIG. 8 is another exemplary functional block diagram of memoryconfigured to store instructions for performing one or more operationsconsistent with the disclosed embodiments.

FIG. 9A is an illustration of an example situation in which a vehiclemay detect and respond to a cut in, consistent with the disclosedembodiments.

FIGS. 9B-9E illustrate example predetermined cut in sensitivity changefactors, consistent with the disclosed embodiments.

FIG. 10 is an illustration of an example situation in which a vehiclemay engage in altruistic behavior, consistent with the disclosedembodiments.

FIG. 11 is a flowchart showing an exemplary process for vehicle cut indetection and response, consistent with disclosed embodiments.

FIG. 12 is a flowchart showing an exemplary process 1200 for navigatingwhile taking into account altruistic behavioral considerations,consistent with disclosed embodiments.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples. Instead, the proper scope is defined by the appended claims.

Autonomous Vehicle Overview

As used throughout this disclosure, the term “autonomous vehicle” refersto a vehicle capable of implementing at least one navigational changewithout driver input. A “navigational change” refers to a change in oneor more of steering, braking, or acceleration of the vehicle. To beautonomous, a vehicle need not be fully automatic (e.g., fully operationwithout a driver or without driver input). Rather, an autonomous vehicleincludes those that can operate under driver control during certain timeperiods and without driver control during other time periods. Autonomousvehicles may also include vehicles that control only some aspects ofvehicle navigation, such as steering (e.g., to maintain a vehicle coursebetween vehicle lane constraints), but may leave other aspects to thedriver (e.g., braking). In some cases, autonomous vehicles may handlesome or all aspects of braking, speed control, and/or steering of thevehicle.

As human drivers typically rely on visual cues and observations in orderto control a vehicle, transportation infrastructures are builtaccordingly, with lane markings, traffic signs, and traffic lights alldesigned to provide visual information to drivers. In view of thesedesign characteristics of transportation infrastructures, an autonomousvehicle may include a camera and a processing unit that analyzes visualinformation captured from the environment of the vehicle. The visualinformation may include, for example, components of the transportationinfrastructure (e.g., lane markings, traffic signs, traffic lights,etc.) that are observable by drivers and other obstacles (e.g., othervehicles, pedestrians, debris, etc.). Additionally, an autonomousvehicle may also use stored information, such as information thatprovides a model of the vehicle's environment when navigating. Forexample, the vehicle may use GPS data, sensor data (e.g., from anaccelerometer, a speed sensor, a suspension sensor, etc.), and/or othermap data to provide information related to its environment while it istraveling, and the vehicle (as well as other vehicles) may use theinformation to localize itself on the model.

System Overview

FIG. 1 is a block diagram representation of a system 100 consistent withthe exemplary disclosed embodiments. System 100 may include variouscomponents depending on the requirements of a particular implementation.In some embodiments, system 100 may include a processing unit 110, animage acquisition unit 120, a position sensor 130, one or more memoryunits 140, 150, a map database 160, a user interface 170, and a wirelesstransceiver 172. Processing unit 110 may include one or more processingdevices. In some embodiments, processing unit 110 may include anapplications processor 180, an image processor 190, or any othersuitable processing device. Similarly, image acquisition unit 120 mayinclude any number of image acquisition devices and components dependingon the requirements of a particular application. In some embodiments,image acquisition unit 120 may include one or more image capture devices(e.g., cameras), such as image capture device 122, image capture device124, and image capture device 126. System 100 may also include a datainterface 128 communicatively connecting processing unit 110 to imageacquisition unit 120. For example, data interface 128 may include anywired and/or wireless link or links for transmitting image data acquiredby image accusation unit 120 to processing unit 110.

Wireless transceiver 172 may include one or more devices configured toexchange transmissions over an air interface to one or more networks(e.g., cellular, the Internet, etc.) by use of a radio frequency,infrared frequency, magnetic field, or an electric field. Wirelesstransceiver 172 may use any known standard to transmit and/or receivedata (e.g., Wi- Fi, Bluetooth®, Bluetooth Smart, 802.15.4, ZigBee,etc.).

Both applications processor 180 and image processor 190 may includevarious types of processing devices. For example, either or both ofapplications processor 180 and image processor 190 may include amicroprocessor, preprocessors (such as an image preprocessor), graphicsprocessors, a central processing unit (CPU), support circuits, digitalsignal processors, integrated circuits, memory, or any other types ofdevices suitable for running applications and for image processing andanalysis. In some embodiments, applications processor 180 and/or imageprocessor 190 may include any type of single or multi-core processor,mobile device microcontroller, central processing unit, etc. Variousprocessing devices may be used, including, for example, processorsavailable from manufacturers such as Intel®, AMD®, etc., and may includevarious architectures (e.g., x86 processor, ARM®, etc.).

In some embodiments, applications processor 180 and/or image processor190 may include any of the EyeQ series of processor chips available fromMobileye®. These processor designs each include multiple processingunits with local memory and instruction sets. Such processors mayinclude video inputs for receiving image data from multiple imagesensors and may also include video out capabilities. In one example, theEyeQ2® uses 90 nm-micron technology operating at 332 Mhz. The EyeQ2®architecture consists of two floating point, hyper-thread 32-bit RISCCPUs (MIPS32® 34K® cores), five Vision Computing Engines (VCE), threeVector Microcode Processors (VMP®), Denali 64-bit Mobile DDR Controller,128-bit internal Sonics Interconnect, dual 16-bit Video input and 18-bitVideo output controllers, 16 channels DMA and several peripherals. TheMIPS34K CPU manages the five VCEs, three VMP™ and the DMA, the secondMIPS34K CPU and the multi-channel DMA as well as the other peripherals.The five VCEs, three VMP® and the MIPS34K CPU can perform intensivevision computations required by multi-function bundle applications. Inanother example, the EyeQ3®, which is a third generation processor andis six times more powerful that the EyeQ2®, may be used in the disclosedembodiments.

Any of the processing devices disclosed herein may be configured toperform certain functions. Configuring a processing device, such as anyof the described EyeQ processors or other controller or microprocessor,to perform certain functions may include programming of computerexecutable instructions and making those instructions available to theprocessing device for execution during operation of the processingdevice. In some embodiments, configuring a processing device may includeprogramming the processing device directly with architecturalinstructions. In other embodiments, configuring a processing device mayinclude storing executable instructions on a memory that is accessibleto the processing device during operation. For example, the processingdevice may access the memory to obtain and execute the storedinstructions during operation.

While FIG. 1 depicts two separate processing devices included inprocessing unit 110, more or fewer processing devices may be used. Forexample, in some embodiments, a single processing device may be used toaccomplish the tasks of applications processor 180 and image processor190. In other embodiments, these tasks may be performed by more than twoprocessing devices. Further, in some embodiments, system 100 may includeone or more of processing unit 110 without including other components,such as image acquisition unit 120.

Processing unit 110 may comprise various types of devices. For example,processing unit 110 may include various devices, such as a controller,an image preprocessor, a central processing unit (CPU), supportcircuits, digital signal processors, integrated circuits, memory, or anyother types of devices for image processing and analysis. The imagepreprocessor may include a video processor for capturing, digitizing andprocessing the imagery from the image sensors. The CPU may comprise anynumber of microcontrollers or microprocessors. The support circuits maybe any number of circuits generally well known in the art, includingcache, power supply, clock and input-output circuits. The memory maystore software that, when executed by the processor, controls theoperation of the system. The memory may include databases and imageprocessing software. The memory may comprise any number of random accessmemories, read only memories, flash memories, disk drives, opticalstorage, tape storage, removable storage and other types of storage. Inone instance, the memory may be separate from the processing unit 110.In another instance, the memory may be integrated into the processingunit 110.

Each memory 140, 150 may include software instructions that whenexecuted by a processor (e.g., applications processor 180 and/or imageprocessor 190), may control operation of various aspects of system 100.These memory units may include various databases and image processingsoftware. The memory units may include random access memory, read onlymemory, flash memory, disk drives, optical storage, tape storage,removable storage and/or any other types of storage. In someembodiments, memory units 140, 150 may be separate from the applicationsprocessor 180 and/or image processor 190. In other embodiments, thesememory units may be integrated into applications processor 180 and/orimage processor 190.

Position sensor 130 may include any type of device suitable fordetermining a location associated with at least one component of system100. In some embodiments, position sensor 130 may include a GPSreceiver. Such receivers can determine a user position and velocity byprocessing signals broadcasted by global positioning system satellites.Position information from position sensor 130 may be made available toapplications processor 180 and/or image processor 190.

In some embodiments, system 100 may include components such as a speedsensor (e.g., a tachometer) for measuring a speed of vehicle 200 and/oran accelerometer for measuring acceleration of vehicle 200.

User interface 170 may include any device suitable for providinginformation to or for receiving inputs from one or more users of system100. In some embodiments, user interface 170 may include user inputdevices, including, for example, a touchscreen, microphone, keyboard,pointer devices, track wheels, cameras, knobs, buttons, etc. With suchinput devices, a user may be able to provide information inputs orcommands to system 100 by typing instructions or information, providingvoice commands, selecting menu options on a screen using buttons,pointers, or eye-tracking capabilities, or through any other suitabletechniques for communicating information to system 100.

User interface 170 may be equipped with one or more processing devicesconfigured to provide and receive information to or from a user andprocess that information for use by, for example, applications processor180. In some embodiments, such processing devices may executeinstructions for recognizing and tracking eye movements, receiving andinterpreting voice commands, recognizing and interpreting touches and/orgestures made on a touchscreen, responding to keyboard entries or menuselections, etc. In some embodiments, user interface 170 may include adisplay, speaker, tactile device, and/or any other devices for providingoutput information to a user.

Map database 160 may include any type of database for storing map datauseful to system 100. In some embodiments, map database 160 may includedata relating to the position, in a reference coordinate system, ofvarious items, including roads, water features, geographic features,businesses, points of interest, restaurants, gas stations, etc. Mapdatabase 160 may store not only the locations of such items, but alsodescriptors relating to those items, including, for example, namesassociated with any of the stored features. In some embodiments, mapdatabase 160 may be physically located with other components of system100. Alternatively or additionally, map database 160 or a portionthereof may be located remotely with respect to other components ofsystem 100 (e.g., processing unit 110). In such embodiments, informationfrom map database 160 may be downloaded over a wired or wireless dataconnection to a network (e.g., over a cellular network and/or theInternet, etc.).

Image capture devices 122, 124, and 126 may each include any type ofdevice suitable for capturing at least one image from an environment.Moreover, any number of image capture devices may be used to acquireimages for input to the image processor. Some embodiments may includeonly a single image capture device, while other embodiments may includetwo, three, or even four or more image capture devices. Image capturedevices 122, 124, and 126 will be further described with reference toFIGS. 2B-2E, below.

System 100, or various components thereof, may be incorporated intovarious different platforms. In some embodiments, system 100 may beincluded on a vehicle 200, as shown in FIG. 2A. For example, vehicle 200may be equipped with a processing unit 110 and any of the othercomponents of system 100, as described above relative to FIG. 1. Whilein some embodiments vehicle 200 may be equipped with only a single imagecapture device (e.g., camera), in other embodiments, such as thosediscussed in connection with FIGS. 2B-2E, multiple image capture devicesmay be used. For example, either of image capture devices 122 and 124 ofvehicle 200, as shown in FIG. 2A, may be part of an ADAS (AdvancedDriver Assistance Systems) imaging set.

The image capture devices included on vehicle 200 as part of the imageacquisition unit 120 may be positioned at any suitable location. In someembodiments, as shown in FIGS. 2A-2E and 3A-3C, image capture device 122may be located in the vicinity of the rearview mirror. This position mayprovide a line of sight similar to that of the driver of vehicle 200,which may aid in determining what is and is not visible to the driver.Image capture device 122 may be positioned at any location near therearview mirror, but placing image capture device 122 on the driver sideof the mirror may further aid in obtaining images representative of thedriver's field of view and/or line of sight.

Other locations for the image capture devices of image acquisition unit120 may also be used. For example, image capture device 124 may belocated on or in a bumper of vehicle 200. Such a location may beespecially suitable for image capture devices having a wide field ofview. The line of sight of bumper-located image capture devices can bedifferent from that of the driver and, therefore, the bumper imagecapture device and driver may not always see the same objects.

The image capture devices (e.g., image capture devices 122, 124, and126) may also be located in other locations. For example, the imagecapture devices may be located on or in one or both of the side mirrorsof vehicle 200, on the roof of vehicle 200, on the hood of vehicle 200,on the trunk of vehicle 200, on the sides of vehicle 200, mounted on,positioned behind, or positioned in front of any of the windows ofvehicle 200, and mounted in or near light fixtures on the front and/orback of vehicle 200, etc.

In addition to image capture devices, vehicle 200 may include variousother components of system 100. For example, processing unit 110 may beincluded on vehicle 200 either integrated with or separate from anengine control unit (ECU) of the vehicle. Vehicle 200 may also beequipped with a position sensor 130, such as a GPS receiver and may alsoinclude a map database 160 and memory units 140 and 150.

As discussed earlier, wireless transceiver 172 may and/or receive dataover one or more networks (e.g., cellular networks, the Internet, etc.).For example, wireless transceiver 172 may upload data collected bysystem 100 to one or more servers, and download data from the one ormore servers. Via wireless transceiver 172, system 100 may receive, forexample, periodic or on-demand updates to data stored in map database160, memory 140, and/or memory 150. Similarly, wireless transceiver 172may upload any data (e.g., images captured by image acquisition unit120, data received by position sensor 130 or other sensors, vehiclecontrol systems, etc.) from system 100 and/or any data processed byprocessing unit 110 to the one or more servers.

System 100 may upload data to a server (e.g., to the cloud) based on aprivacy level setting. For example, system 100 may implement privacylevel settings to regulate or limit the types of data (includingmetadata) sent to the server that may uniquely identify a vehicle and ordriver/owner of a vehicle. Such settings may be set by user via, forexample, wireless transceiver 172, be initialized by factory defaultsettings, or by data received by wireless transceiver 172.

In some embodiments, system 100 may upload data according to a “high”privacy level, and under setting a setting, system 100 may transmit data(e.g., location information related to a route, captured images, etc.)without any details about the specific vehicle and/or driver/owner.

For example, when uploading data according to a “high” privacy setting,system 100 may not include a vehicle identification number (VIN) or aname of a driver or owner of the vehicle, and may instead transmit data,such as captured images and/or limited location information related to aroute.

Other privacy levels are contemplated as well. For example, system 100may transmit data to a server according to an “intermediate” privacylevel and include additional information not included under a “high”privacy level, such as a make and/or model of a vehicle and/or a vehicletype (e.g., a passenger vehicle, sport utility vehicle, truck, etc.). Insome embodiments, system 100 may upload data according to a “low”privacy level. Under a “low” privacy level setting, system 100 mayupload data and include information sufficient to uniquely identify aspecific vehicle, owner/driver, and/or a portion or entirely of a routetraveled by the vehicle. Such “low” privacy level data may include oneor more of, for example, a VIN, a driver/owner name, an originationpoint of a vehicle prior to departure, an intended destination of thevehicle, a make and/or model of the vehicle, a type of the vehicle, etc.

FIG. 2A is a diagrammatic side view representation of an exemplaryvehicle imaging system consistent with the disclosed embodiments. FIG.2B is a diagrammatic top view illustration of the embodiment shown inFIG. 2A. As illustrated in FIG. 2B, the disclosed embodiments mayinclude a vehicle 200 including in its body a system 100 with a firstimage capture device 122 positioned in the vicinity of the rearviewmirror and/or near the driver of vehicle 200, a second image capturedevice 124 positioned on or in a bumper region (e.g., one of bumperregions 210) of vehicle 200, and a processing unit 110.

As illustrated in FIG. 2C, image capture devices 122 and 124 may both bepositioned in the vicinity of the rearview mirror and/or near the driverof vehicle 200. Additionally, while two image capture devices 122 and124 are shown in FIGS. 2B and 2C, it should be understood that otherembodiments may include more than two image capture devices. Forexample, in the embodiments shown in FIGS. 2D and 2E, first, second, andthird image capture devices 122, 124, and 126, are included in thesystem 100 of vehicle 200.

As illustrated in FIG. 2D, image capture device 122 may be positioned inthe vicinity of the rearview mirror and/or near the driver of vehicle200, and image capture devices 124 and 126 may be positioned on or in abumper region (e.g., one of bumper regions 210) of vehicle 200. And asshown in FIG. 2E, image capture devices 122, 124, and 126 may bepositioned in the vicinity of the rearview mirror and/or near the driverseat of vehicle 200. The disclosed embodiments are not limited to anyparticular number and configuration of the image capture devices, andthe image capture devices may be positioned in any appropriate locationwithin and/or on vehicle 200.

It is to be understood that the disclosed embodiments are not limited tovehicles and could be applied in other contexts. It is also to beunderstood that disclosed embodiments are not limited to a particulartype of vehicle 200 and may be applicable to all types of vehiclesincluding automobiles, trucks, trailers, and other types of vehicles.

The first image capture device 122 may include any suitable type ofimage capture device. Image capture device 122 may include an opticalaxis. In one instance, the image capture device 122 may include anAptina M9V024 WVGA sensor with a global shutter. In other embodiments,image capture device 122 may provide a resolution of 1280×960 pixels andmay include a rolling shutter. Image capture device 122 may includevarious optical elements. In some embodiments one or more lenses may beincluded, for example, to provide a desired focal length and field ofview for the image capture device. In some embodiments, image capturedevice 122 may be associated with a 6mm lens or a 12mm lens. In someembodiments, image capture device 122 may be configured to captureimages having a desired field-of-view (FOV) 202, as illustrated in FIG.2D. For example, image capture device 122 may be configured to have aregular FOV, such as within a range of 40 degrees to 56 degrees,including a 46 degree FOV, 50 degree FOV, 52 degree FOV, or greater.Alternatively, image capture device 122 may be configured to have anarrow FOV in the range of 23 to 40 degrees, such as a 28 degree FOV or36 degree FOV. In addition, image capture device 122 may be configuredto have a wide FOV in the range of 100 to 180 degrees. In someembodiments, image capture device 122 may include a wide angle bumpercamera or one with up to a 180 degree FOV. In some embodiments, imagecapture device 122 may be a 7.2M pixel image capture device with anaspect ratio of about 2:1 (e.g., HxV=3800×1900 pixels) with about 100degree horizontal FOV. Such an image capture device may be used in placeof a three image capture device configuration. Due to significant lensdistortion, the vertical FOV of such an image capture device may besignificantly less than 50 degrees in implementations in which the imagecapture device uses a radially symmetric lens. For example, such a lensmay not be radially symmetric which would allow for a vertical FOVgreater than 50 degrees with 100 degree horizontal FOV.

The first image capture device 122 may acquire a plurality of firstimages relative to a scene associated with vehicle 200. Each of theplurality of first images may be acquired as a series of image scanlines, which may be captured using a rolling shutter. Each scan line mayinclude a plurality of pixels.

The first image capture device 122 may have a scan rate associated withacquisition of each of the first series of image scan lines. The scanrate may refer to a rate at which an image sensor can acquire image dataassociated with each pixel included in a particular scan line.

Image capture devices 122, 124, and 126 may contain any suitable typeand number of image sensors, including CCD sensors or CMOS sensors, forexample. In one embodiment, a CMOS image sensor may be employed alongwith a rolling shutter, such that each pixel in a row is read one at atime, and scanning of the rows proceeds on a row-by-row basis until anentire image frame has been captured. In some embodiments, the rows maybe captured sequentially from top to bottom relative to the frame.

In some embodiments, one or more of the image capture devices (e.g.,image capture devices 122, 124, and 126) disclosed herein may constitutea high resolution imager and may have a resolution greater than 5Mpixel, 7M pixel, 10M pixel, or greater.

The use of a rolling shutter may result in pixels in different rowsbeing exposed and captured at different times, which may cause skew andother image artifacts in the captured image frame. On the other hand,when the image capture device 122 is configured to operate with a globalor synchronous shutter, all of the pixels may be exposed for the sameamount of time and during a common exposure period. As a result, theimage data in a frame collected from a system employing a global shutterrepresents a snapshot of the entire FOV (such as FOV 202) at aparticular time. In contrast, in a rolling shutter application, each rowin a frame is exposed and data is capture at different times. Thus,moving objects may appear distorted in an image capture device having arolling shutter. This phenomenon will be described in greater detailbelow.

The second image capture device 124 and the third image capturing device126 may be any type of image capture device. Like the first imagecapture device 122, each of image capture devices 124 and 126 mayinclude an optical axis. In one embodiment, each of image capturedevices 124 and 126 may include an Aptina M9V024 WVGA sensor with aglobal shutter. Alternatively, each of image capture devices 124 and 126may include a rolling shutter. Like image capture device 122, imagecapture devices 124 and 126 may be configured to include various lensesand optical elements. In some embodiments, lenses associated with imagecapture devices 124 and 126 may provide FOVs (such as FOVs 204 and 206)that are the same as, or narrower than, a FOV (such as FOV 202)associated with image capture device 122. For example, image capturedevices 124 and 126 may have FOVs of 40 degrees, 30 degrees, 26 degrees,23 degrees, 20 degrees, or less.

Image capture devices 124 and 126 may acquire a plurality of second andthird images relative to a scene associated with vehicle 200. Each ofthe plurality of second and third images may be acquired as a second andthird series of image scan lines, which may be captured using a rollingshutter. Each scan line or row may have a plurality of pixels. Imagecapture devices 124 and 126 may have second and third scan ratesassociated with acquisition of each of image scan lines included in thesecond and third series.

Each image capture device 122, 124, and 126 may be positioned at anysuitable position and orientation relative to vehicle 200. The relativepositioning of the image capture devices 122, 124, and 126 may beselected to aid in fusing together the information acquired from theimage capture devices. For example, in some embodiments, a FOV (such asFOV 204) associated with image capture device 124 may overlap partiallyor fully with a FOV (such as FOV 202) associated with image capturedevice 122 and a FOV (such as FOV 206) associated with image capturedevice 126.

Image capture devices 122, 124, and 126 may be located on vehicle 200 atany suitable relative heights. In one instance, there may be a heightdifference between the image capture devices 122, 124, and 126, whichmay provide sufficient parallax information to enable stereo analysis.For example, as shown in FIG. 2A, the two image capture devices 122 and124 are at different heights. There may also be a lateral displacementdifference between image capture devices 122, 124, and 126, givingadditional parallax information for stereo analysis by processing unit110, for example. The difference in the lateral displacement may bedenoted by d_(x), as shown in FIGS. 2C and 2D. In some embodiments, foreor aft displacement (e.g., range displacement) may exist between imagecapture devices 122, 124, and 126. For example, image capture device 122may be located 0.5 to 2 meters or more behind image capture device 124and/or image capture device 126. This type of displacement may enableone of the image capture devices to cover potential blind spots of theother image capture device(s).

Image capture devices 122 may have any suitable resolution capability(e.g., number of pixels associated with the image sensor), and theresolution of the image sensor(s) associated with the image capturedevice 122 may be higher, lower, or the same as the resolution of theimage sensor(s) associated with image capture devices 124 and 126. Insome embodiments, the image sensor(s) associated with image capturedevice 122 and/or image capture devices 124 and 126 may have aresolution of 640×480, 1024×768, 1280×960, or any other suitableresolution.

The frame rate (e.g., the rate at which an image capture device acquiresa set of pixel data of one image frame before moving on to capture pixeldata associated with the next image frame) may be controllable. Theframe rate associated with image capture device 122 may be higher,lower, or the same as the frame rate associated with image capturedevices 124 and 126. The frame rate associated with image capturedevices 122, 124, and 126 may depend on a variety of factors that mayaffect the timing of the frame rate. For example, one or more of imagecapture devices 122, 124, and 126 may include a selectable pixel delayperiod imposed before or after acquisition of image data associated withone or more pixels of an image sensor in image capture device 122, 124,and/or 126. Generally, image data corresponding to each pixel may beacquired according to a clock rate for the device (e.g., one pixel perclock cycle). Additionally, in embodiments including a rolling shutter,one or more of image capture devices 122, 124, and 126 may include aselectable horizontal blanking period imposed before or afteracquisition of image data associated with a row of pixels of an imagesensor in image capture device 122, 124, and/or 126. Further, one ormore of image capture devices 122, 124, and/or 126 may include aselectable vertical blanking period imposed before or after acquisitionof image data associated with an image frame of image capture device122, 124, and 126.

These timing controls may enable synchronization of frame ratesassociated with image capture devices 122, 124, and 126, even where theline scan rates of each are different. Additionally, as will bediscussed in greater detail below, these selectable timing controls,among other factors (e.g., image sensor resolution, maximum line scanrates, etc.) may enable synchronization of image capture from an areawhere the FOV of image capture device 122 overlaps with one or more FOVsof image capture devices 124 and 126, even where the field of view ofimage capture device 122 is different from the FOVs of image capturedevices 124 and 126.

Frame rate timing in image capture device 122, 124, and 126 may dependon the resolution of the associated image sensors. For example, assumingsimilar line scan rates for both devices, if one device includes animage sensor having a resolution of 640×480 and another device includesan image sensor with a resolution of 1280×960, then more time will berequired to acquire a frame of image data from the sensor having thehigher resolution.

Another factor that may affect the timing of image data acquisition inimage capture devices 122, 124, and 126 is the maximum line scan rate.For example, acquisition of a row of image data from an image sensorincluded in image capture device 122, 124, and 126 will require someminimum amount of time. Assuming no pixel delay periods are added, thisminimum amount of time for acquisition of a row of image data will berelated to the maximum line scan rate for a particular device. Devicesthat offer higher maximum line scan rates have the potential to providehigher frame rates than devices with lower maximum line scan rates. Insome embodiments, one or more of image capture devices 124 and 126 mayhave a maximum line scan rate that is higher than a maximum line scanrate associated with image capture device 122. In some embodiments, themaximum line scan rate of image capture device 124 and/or 126 may be1.25, 1.5, 1.75, or 2 times or more than a maximum line scan rate ofimage capture device 122.

In another embodiment, image capture devices 122, 124, and 126 may havethe same maximum line scan rate, but image capture device 122 may beoperated at a scan rate less than or equal to its maximum scan rate. Thesystem may be configured such that one or more of image capture devices124 and 126 operate at a line scan rate that is equal to the line scanrate of image capture device 122. In other instances, the system may beconfigured such that the line scan rate of image capture device 124and/or image capture device 126 may be 1.25, 1.5, 1.75, or 2 times ormore than the line scan rate of image capture device 122.

In some embodiments, image capture devices 122, 124, and 126 may beasymmetric. That is, they may include cameras having different fields ofview (FOV) and focal lengths. The fields of view of image capturedevices 122, 124, and 126 may include any desired area relative to anenvironment of vehicle 200, for example. In some embodiments, one ormore of image capture devices 122, 124, and 126 may be configured toacquire image data from an environment in front of vehicle 200, behindvehicle 200, to the sides of vehicle 200, or combinations thereof

Further, the focal length associated with each image capture device 122,124, and/or 126 may be selectable (e.g., by inclusion of appropriatelenses etc.) such that each device acquires images of objects at adesired distance range relative to vehicle 200. For example, in someembodiments image capture devices 122, 124, and 126 may acquire imagesof close-up objects within a few meters from the vehicle. Image capturedevices 122, 124, and 126 may also be configured to acquire images ofobjects at ranges more distant from the vehicle (e.g., 25 m, 50 m, 100m, 150 m, or more). Further, the focal lengths of image capture devices122, 124, and 126 may be selected such that one image capture device(e.g., image capture device 122) can acquire images of objectsrelatively close to the vehicle (e.g., within 10 m or within 20 m) whilethe other image capture devices (e.g., image capture devices 124 and126) can acquire images of more distant objects (e.g., greater than 20m, 50 m, 100 m, 150 m, etc.) from vehicle 200.

According to some embodiments, the FOV of one or more image capturedevices 122, 124, and 126 may have a wide angle. For example, it may beadvantageous to have a FOV of 140 degrees, especially for image capturedevices 122, 124, and 126 that may be used to capture images of the areain the vicinity of vehicle 200. For example, image capture device 122may be used to capture images of the area to the right or left ofvehicle 200 and, in such embodiments, it may be desirable for imagecapture device 122 to have a wide FOV (e.g., at least 140 degrees).

The field of view associated with each of image capture devices 122,124, and 126 may depend on the respective focal lengths. For example, asthe focal length increases, the corresponding field of view decreases.

Image capture devices 122, 124, and 126 may be configured to have anysuitable fields of view. In one particular example, image capture device122 may have a horizontal FOV of 46 degrees, image capture device 124may have a horizontal FOV of 23 degrees, and image capture device 126may have a horizontal FOV in between 23 and 46 degrees. In anotherinstance, image capture device 122 may have a horizontal FOV of 52degrees, image capture device 124 may have a horizontal FOV of 26degrees, and image capture device 126 may have a horizontal FOV inbetween 26 and 52 degrees. In some embodiments, a ratio of the FOV ofimage capture device 122 to the FOVs of image capture device 124 and/orimage capture device 126 may vary from 1.5 to 2.0. In other embodiments,this ratio may vary between 1.25 and 2.25.

System 100 may be configured so that a field of view of image capturedevice 122 overlaps, at least partially or fully, with a field of viewof image capture device 124 and/or image capture device 126. In someembodiments, system 100 may be configured such that the fields of viewof image capture devices 124 and 126, for example, fall within (e.g.,are narrower than) and share a common center with the field of view ofimage capture device 122. In other embodiments, the image capturedevices 122, 124, and 126 may capture adjacent FOVs or may have partialoverlap in their FOVs. In some embodiments, the fields of view of imagecapture devices 122, 124, and 126 may be aligned such that a center ofthe narrower FOV image capture devices 124 and/or 126 may be located ina lower half of the field of view of the wider FOV device 122.

FIG. 2F is a diagrammatic representation of exemplary vehicle controlsystems, consistent with the disclosed embodiments. As indicated in FIG.2F, vehicle 200 may include throttling system 220, braking system 230,and steering system 240. System 100 may provide inputs (e.g., controlsignals) to one or more of throttling system 220, braking system 230,and steering system 240 over one or more data links (e.g., any wiredand/or wireless link or links for transmitting data). For example, basedon analysis of images acquired by image capture devices 122, 124, and/or126, system 100 may provide control signals to one or more of throttlingsystem 220, braking system 230, and steering system 240 to navigatevehicle 200 (e.g., by causing an acceleration, a turn, a lane shift,etc.). Further, system 100 may receive inputs from one or more ofthrottling system 220, braking system 230, and steering system 24indicating operating conditions of vehicle 200 (e.g., speed, whethervehicle 200 is braking and/or turning, etc.). Further details areprovided in connection with FIGS. 4-7, below.

As shown in FIG. 3A, vehicle 200 may also include a user interface 170for interacting with a driver or a passenger of vehicle 200. Forexample, user interface 170 in a vehicle application may include a touchscreen 320, knobs 330, buttons 340, and a microphone 350. A driver orpassenger of vehicle 200 may also use handles (e.g., located on or nearthe steering column of vehicle 200 including, for example, turn signalhandles), buttons (e.g., located on the steering wheel of vehicle 200),and the like, to interact with system 100. In some embodiments,microphone 350 may be positioned adjacent to a rearview mirror 310.Similarly, in some embodiments, image capture device 122 may be locatednear rearview mirror 310. In some embodiments, user interface 170 mayalso include one or more speakers 360 (e.g., speakers of a vehicle audiosystem). For example, system 100 may provide various notifications(e.g., alerts) via speakers 360.

FIGS. 3B-3D are illustrations of an exemplary camera mount 370configured to be positioned behind a rearview mirror (e.g., rearviewmirror 310) and against a vehicle windshield, consistent with disclosedembodiments. As shown in FIG. 3B, camera mount 370 may include imagecapture devices 122, 124, and 126. Image capture devices 124 and 126 maybe positioned behind a glare shield 380, which may be flush against thevehicle windshield and include a composition of film and/oranti-reflective materials. For example, glare shield 380 may bepositioned such that it aligns against a vehicle windshield having amatching slope. In some embodiments, each of image capture devices 122,124, and 126 may be positioned behind glare shield 380, as depicted, forexample, in FIG. 3D. The disclosed embodiments are not limited to anyparticular configuration of image capture devices 122, 124, and 126,camera mount 370, and glare shield 380. FIG. 3C is an illustration ofcamera mount 370 shown in FIG. 3B from a front perspective.

As will be appreciated by a person skilled in the art having the benefitof this disclosure, numerous variations and/or modifications may be madeto the foregoing disclosed embodiments. For example, not all componentsare essential for the operation of system 100. Further, any componentmay be located in any appropriate part of system 100 and the componentsmay be rearranged into a variety of configurations while providing thefunctionality of the disclosed embodiments. Therefore, the foregoingconfigurations are examples and, regardless of the configurationsdiscussed above, system 100 can provide a wide range of functionality toanalyze the surroundings of vehicle 200 and navigate vehicle 200 inresponse to the analysis.

As discussed below in further detail and consistent with variousdisclosed embodiments, system 100 may provide a variety of featuresrelated to autonomous driving and/or driver assist technology. Forexample, system 100 may analyze image data, position data (e.g., GPSlocation information), map data, speed data, and/or data from sensorsincluded in vehicle 200. System 100 may collect the data for analysisfrom, for example, image acquisition unit 120, position sensor 130, andother sensors. Further, system 100 may analyze the collected data todetermine whether or not vehicle 200 should take a certain action, andthen automatically take the determined action without humanintervention. For example, when vehicle 200 navigates without humanintervention, system 100 may automatically control the braking,acceleration, and/or steering of vehicle 200 (e.g., by sending controlsignals to one or more of throttling system 220, braking system 230, andsteering system 240). Further, system 100 may analyze the collected dataand issue warnings and/or alerts to vehicle occupants based on theanalysis of the collected data. Additional details regarding the variousembodiments that are provided by system 100 are provided below.

Forward-Facing Multi-Imaging System

As discussed above, system 100 may provide drive assist functionalitythat uses a multi-camera system. The multi-camera system may use one ormore cameras facing in the forward direction of a vehicle. In otherembodiments, the multi-camera system may include one or more camerasfacing to the side of a vehicle or to the rear of the vehicle. In oneembodiment, for example, system 100 may use a two-camera imaging system,where a first camera and a second camera (e.g., image capture devices122 and 124) may be positioned at the front and/or the sides of avehicle (e.g., vehicle 200). The first camera may have a field of viewthat is greater than, less than, or partially overlapping with, thefield of view of the second camera. In addition, the first camera may beconnected to a first image processor to perform monocular image analysisof images provided by the first camera, and the second camera may beconnected to a second image processor to perform monocular imageanalysis of images provided by the second camera. The outputs (e.g.,processed information) of the first and second image processors may becombined. In some embodiments, the second image processor may receiveimages from both the first camera and second camera to perform stereoanalysis. In another embodiment, system 100 may use a three-cameraimaging system where each of the cameras has a different field of view.Such a system may, therefore, make decisions based on informationderived from objects located at varying distances both forward and tothe sides of the vehicle. References to monocular image analysis mayrefer to instances where image analysis is performed based on imagescaptured from a single point of view (e.g., from a single camera).Stereo image analysis may refer to instances where image analysis isperformed based on two or more images captured with one or morevariations of an image capture parameter. For example, captured imagessuitable for performing stereo image analysis may include imagescaptured: from two or more different positions, from different fields ofview, using different focal lengths, along with parallax information,etc.

For example, in one embodiment, system 100 may implement a three cameraconfiguration using image capture devices 122-126. In such aconfiguration, image capture device 122 may provide a narrow field ofview (e.g., 34 degrees, or other values selected from a range of about20 to 45 degrees, etc.), image capture device 124 may provide a widefield of view (e.g., 150 degrees or other values selected from a rangeof about 100 to about 180 degrees), and image capture device 126 mayprovide an intermediate field of view (e.g., 46 degrees or other valuesselected from a range of about 35 to about 60 degrees). In someembodiments, image capture device 126 may act as a main or primarycamera. Image capture devices 122-126 may be positioned behind rearviewmirror 310 and positioned substantially side-by-side (e.g., 6 cm apart).Further, in some embodiments, as discussed above, one or more of imagecapture devices 122-126 may be mounted behind glare shield 380 that isflush with the windshield of vehicle 200. Such shielding may act tominimize the impact of any reflections from inside the car on imagecapture devices 122-126.

In another embodiment, as discussed above in connection with FIGS. 3Band 3C, the wide field of view camera (e.g., image capture device 124 inthe above example) may be mounted lower than the narrow and main fieldof view cameras (e.g., image devices 122 and 126 in the above example).This configuration may provide a free line of sight from the wide fieldof view camera. To reduce reflections, the cameras may be mounted closeto the windshield of vehicle 200, and may include polarizers on thecameras to damp reflected light.

A three camera system may provide certain performance characteristics.For example, some embodiments may include an ability to validate thedetection of objects by one camera based on detection results fromanother camera. In the three camera configuration discussed above,processing unit 110 may include, for example, three processing devices(e.g., three EyeQ series of processor chips, as discussed above), witheach processing device dedicated to processing images captured by one ormore of image capture devices 122-126.

In a three camera system, a first processing device may receive imagesfrom both the main camera and the narrow field of view camera, andperform vision processing of the narrow FOV camera to, for example,detect other vehicles, pedestrians, lane marks, traffic signs, trafficlights, and other road objects. Further, the first processing device maycalculate a disparity of pixels between the images from the main cameraand the narrow camera and create a 3D reconstruction of the environmentof vehicle 200. The first processing device may then combine the 3Dreconstruction with 3D map data or with 3D information calculated basedon information from another camera.

The second processing device may receive images from the main camera andperform vision processing to detect other vehicles, pedestrians, lanemarks, traffic signs, traffic lights, and other road objects.Additionally, the second processing device may calculate a cameradisplacement and, based on the displacement, calculate a disparity ofpixels between successive images and create a 3D reconstruction of thescene (e.g., a structure from motion). The second processing device maysend the structure from motion based 3D reconstruction to the firstprocessing device to be combined with the stereo 3D images.

The third processing device may receive images from the wide FOV cameraand process the images to detect vehicles, pedestrians, lane marks,traffic signs, traffic lights, and other road objects. The thirdprocessing device may further execute additional processing instructionsto analyze images to identify objects moving in the image, such asvehicles changing lanes, pedestrians, etc.

In some embodiments, having streams of image-based information capturedand processed independently may provide an opportunity for providingredundancy in the system. Such redundancy may include, for example,using a first image capture device and the images processed from thatdevice to validate and/or supplement information obtained by capturingand processing image information from at least a second image capturedevice.

In some embodiments, system 100 may use two image capture devices (e.g.,image capture devices 122 and 124) in providing navigation assistancefor vehicle 200 and use a third image capture device (e.g., imagecapture device 126) to provide redundancy and validate the analysis ofdata received from the other two image capture devices. For example, insuch a configuration, image capture devices 122 and 124 may provideimages for stereo analysis by system 100 for navigating vehicle 200,while image capture device 126 may provide images for monocular analysisby system 100 to provide redundancy and validation of informationobtained based on images captured from image capture device 122 and/orimage capture device 124. That is, image capture device 126 (and acorresponding processing device) may be considered to provide aredundant sub-system for providing a check on the analysis derived fromimage capture devices 122 and 124 (e.g., to provide an automaticemergency braking (AEB) system).

One of skill in the art will recognize that the above cameraconfigurations, camera placements, number of cameras, camera locations,etc., are examples only. These components and others described relativeto the overall system may be assembled and used in a variety ofdifferent configurations without departing from the scope of thedisclosed embodiments. Further details regarding usage of a multi-camerasystem to provide driver assist and/or autonomous vehicle functionalityfollow below.

FIG. 4 is an exemplary functional block diagram of memory 140 and/or150, which may be stored/programmed with instructions for performing oneor more operations consistent with the disclosed embodiments. Althoughthe following refers to memory 140, one of skill in the art willrecognize that instructions may be stored in memory 140 and/or 150.

As shown in FIG. 4, memory 140 may store a monocular image analysismodule 402, a stereo image analysis module 404, a velocity andacceleration module 406, and a navigational response module 408. Thedisclosed embodiments are not limited to any particular configuration ofmemory 140. Further, applications processor 180 and/or image processor190 may execute the instructions stored in any of modules 402-408included in memory 140. One of skill in the art will understand thatreferences in the following discussions to processing unit 110 may referto applications processor 180 and image processor 190 individually orcollectively. Accordingly, steps of any of the following processes maybe performed by one or more processing devices.

In one embodiment, monocular image analysis module 402 may storeinstructions (such as computer vision software) which, when executed byprocessing unit 110, performs monocular image analysis of a set ofimages acquired by one of image capture devices 122, 124, and 126. Insome embodiments, processing unit 110 may combine information from a setof images with additional sensory information (e.g., information fromradar) to perform the monocular image analysis. As described inconnection with FIGS. 5A-5D below, monocular image analysis module 402may include instructions for detecting a set of features within the setof images, such as lane markings, vehicles, pedestrians, road signs,highway exit ramps, traffic lights, hazardous objects, and any otherfeature associated with an environment of a vehicle. Based on theanalysis, system 100 (e.g., via processing unit 110) may cause one ormore navigational responses in vehicle 200, such as a turn, a laneshift, a change in acceleration, and the like, as discussed below inconnection with navigational response module 408.

In one embodiment, stereo image analysis module 404 may storeinstructions (such as computer vision software) which, when executed byprocessing unit 110, performs stereo image analysis of first and secondsets of images acquired by a combination of image capture devicesselected from any of image capture devices 122, 124, and 126. In someembodiments, processing unit 110 may combine information from the firstand second sets of images with additional sensory information (e.g.,information from radar) to perform the stereo image analysis. Forexample, stereo image analysis module 404 may include instructions forperforming stereo image analysis based on a first set of images acquiredby image capture device 124 and a second set of images acquired by imagecapture device 126. As described in connection with FIG. 6 below, stereoimage analysis module 404 may include instructions for detecting a setof features within the first and second sets of images, such as lanemarkings, vehicles, pedestrians, road signs, highway exit ramps, trafficlights, hazardous objects, and the like. Based on the analysis,processing unit 110 may cause one or more navigational responses invehicle 200, such as a turn, a lane shift, a change in acceleration, andthe like, as discussed below in connection with navigational responsemodule 408.

In one embodiment, velocity and acceleration module 406 may storesoftware configured to analyze data received from one or more computingand electromechanical devices in vehicle 200 that are configured tocause a change in velocity and/or acceleration of vehicle 200. Forexample, processing unit 110 may execute instructions associated withvelocity and acceleration module 406 to calculate a target speed forvehicle 200 based on data derived from execution of monocular imageanalysis module 402 and/or stereo image analysis module 404. Such datamay include, for example, a target position, velocity, and/oracceleration, the position and/or speed of vehicle 200 relative to anearby vehicle, pedestrian, or road object, position information forvehicle 200 relative to lane markings of the road, and the like. Inaddition, processing unit 110 may calculate a target speed for vehicle200 based on sensory input (e.g., information from radar) and input fromother systems of vehicle 200, such as throttling system 220, brakingsystem 230, and/or steering system 240 of vehicle 200. Based on thecalculated target speed, processing unit 110 may transmit electronicsignals to throttling system 220, braking system 230, and/or steeringsystem 240 of vehicle 200 to trigger a change in velocity and/oracceleration by, for example, physically depressing the brake or easingup off the accelerator of vehicle 200.

In one embodiment, navigational response module 408 may store softwareexecutable by processing unit 110 to determine a desired navigationalresponse based on data derived from execution of monocular imageanalysis module 402 and/or stereo image analysis module 404. Such datamay include position and speed information associated with nearbyvehicles, pedestrians, and road objects, target position information forvehicle 200, and the like. Additionally, in some embodiments, thenavigational response may be based (partially or fully) on map data, apredetermined position of vehicle 200, and/or a relative velocity or arelative acceleration between vehicle 200 and one or more objectsdetected from execution of monocular image analysis module 402 and/orstereo image analysis module 404. Navigational response module 408 mayalso determine a desired navigational response based on sensory input(e.g., information from radar) and inputs from other systems of vehicle200, such as throttling system 220, braking system 230, and steeringsystem 240 of vehicle 200. Based on the desired navigational response,processing unit 110 may transmit electronic signals to throttling system220, braking system 230, and steering system 240 of vehicle 200 totrigger a desired navigational response by, for example, turning thesteering wheel of vehicle 200 to achieve a rotation of a predeterminedangle. In some embodiments, processing unit 110 may use the output ofnavigational response module 408 (e.g., the desired navigationalresponse) as an input to execution of velocity and acceleration module406 for calculating a change in speed of vehicle 200.

FIG. 5A is a flowchart showing an exemplary process 500A for causing oneor more navigational responses based on monocular image analysis,consistent with disclosed embodiments. At step 510, processing unit 110may receive a plurality of images via data interface 128 betweenprocessing unit 110 and image acquisition unit 120. For instance, acamera included in image acquisition unit 120 (such as image capturedevice 122 having field of view 202) may capture a plurality of imagesof an area forward of vehicle 200 (or to the sides or rear of a vehicle,for example) and transmit them over a data connection (e.g., digital,wired, USB, wireless, Bluetooth, etc.) to processing unit 110.Processing unit 110 may execute monocular image analysis module 402 toanalyze the plurality of images at step 520, as described in furtherdetail in connection with FIGS.

5B-5D below. By performing the analysis, processing unit 110 may detecta set of features within the set of images, such as lane markings,vehicles, pedestrians, road signs, highway exit ramps, traffic lights,and the like.

Processing unit 110 may also execute monocular image analysis module 402to detect various road hazards at step 520, such as, for example, partsof a truck tire, fallen road signs, loose cargo, small animals, and thelike. Road hazards may vary in structure, shape, size, and color, whichmay make detection of such hazards more challenging. In someembodiments, processing unit 110 may execute monocular image analysismodule 402 to perform multi-frame analysis on the plurality of images todetect road hazards. For example, processing unit 110 may estimatecamera motion between consecutive image frames and calculate thedisparities in pixels between the frames to construct a 3D-map of theroad. Processing unit 110 may then use the 3D-map to detect the roadsurface, as well as hazards existing above the road surface.

At step 530, processing unit 110 may execute navigational responsemodule 408 to cause one or more navigational responses in vehicle 200based on the analysis performed at step 520 and the techniques asdescribed above in connection with FIG. 4. Navigational responses mayinclude, for example, a turn, a lane shift, a change in acceleration,and the like. In some embodiments, processing unit 110 may use dataderived from execution of velocity and acceleration module 406 to causethe one or more navigational responses. Additionally, multiplenavigational responses may occur simultaneously, in sequence, or anycombination thereof. For instance, processing unit 110 may cause vehicle200 to shift one lane over and then accelerate by, for example,sequentially transmitting control signals to steering system 240 andthrottling system 220 of vehicle 200. Alternatively, processing unit 110may cause vehicle 200 to brake while at the same time shifting lanes by,for example, simultaneously transmitting control signals to brakingsystem 230 and steering system 240 of vehicle 200.

FIG. 5B is a flowchart showing an exemplary process 500B for detectingone or more vehicles and/or pedestrians in a set of images, consistentwith disclosed embodiments. Processing unit 110 may execute monocularimage analysis module 402 to implement process 500B. At step 540,processing unit 110 may determine a set of candidate objectsrepresenting possible vehicles and/or pedestrians. For example,processing unit 110 may scan one or more images, compare the images toone or more predetermined patterns, and identify within each imagepossible locations that may contain objects of interest (e.g., vehicles,pedestrians, or portions thereof). The predetermined patterns may bedesigned in such a way to achieve a high rate of “false hits” and a lowrate of “misses.” For example, processing unit 110 may use a lowthreshold of similarity to predetermined patterns for identifyingcandidate objects as possible vehicles or pedestrians. Doing so mayallow processing unit 110 to reduce the probability of missing (e.g.,not identifying) a candidate object representing a vehicle orpedestrian.

At step 542, processing unit 110 may filter the set of candidate objectsto exclude certain candidates (e.g., irrelevant or less relevantobjects) based on classification criteria. Such criteria may be derivedfrom various properties associated with object types stored in adatabase (e.g., a database stored in memory 140). Properties may includeobject shape, dimensions, texture, position (e.g., relative to vehicle200), and the like. Thus, processing unit 110 may use one or more setsof criteria to reject false candidates from the set of candidateobjects.

At step 544, processing unit 110 may analyze multiple frames of imagesto determine whether objects in the set of candidate objects representvehicles and/or pedestrians. For example, processing unit 110 may tracka detected candidate object across consecutive frames and accumulateframe-by-frame data associated with the detected object (e.g., size,position relative to vehicle 200, etc.). Additionally, processing unit110 may estimate parameters for the detected object and compare theobject's frame-by-frame position data to a predicted position.

At step 546, processing unit 110 may construct a set of measurements forthe detected objects. Such measurements may include, for example,position, velocity, and acceleration values (relative to vehicle 200)associated with the detected objects. In some embodiments, processingunit 110 may construct the measurements based on estimation techniquesusing a series of time-based observations such as Kalman filters orlinear quadratic estimation (LQE), and/or based on available modelingdata for different object types (e.g., cars, trucks, pedestrians,bicycles, road signs, etc.). The Kalman filters may be based on ameasurement of an object's scale, where the scale measurement isproportional to a time to collision (e.g., the amount of time forvehicle 200 to reach the object). Thus, by performing steps 540-546,processing unit 110 may identify vehicles and pedestrians appearingwithin the set of captured images and derive information (e.g.,position, speed, size) associated with the vehicles and pedestrians.Based on the identification and the derived information, processing unit110 may cause one or more navigational responses in vehicle 200, asdescribed in connection with FIG. 5A, above.

At step 548, processing unit 110 may perform an optical flow analysis ofone or more images to reduce the probabilities of detecting a “falsehit” and missing a candidate object that represents a vehicle orpedestrian. The optical flow analysis may refer to, for example,analyzing motion patterns relative to vehicle 200 in the one or moreimages associated with other vehicles and pedestrians, and that aredistinct from road surface motion. Processing unit 110 may calculate themotion of candidate objects by observing the different positions of theobjects across multiple image frames, which are captured at differenttimes. Processing unit 110 may use the position and time values asinputs into mathematical models for calculating the motion of thecandidate objects. Thus, optical flow analysis may provide anothermethod of detecting vehicles and pedestrians that are nearby vehicle200. Processing unit 110 may perform optical flow analysis incombination with steps 540-546 to provide redundancy for detectingvehicles and pedestrians and increase the reliability of system 100.

FIG. 5C is a flowchart showing an exemplary process 500C for detectingroad marks and/or lane geometry information in a set of images,consistent with disclosed embodiments. Processing unit 110 may executemonocular image analysis module 402 to implement process 500C. At step550, processing unit 110 may detect a set of objects by scanning one ormore images. To detect segments of lane markings, lane geometryinformation, and other pertinent road marks, processing unit 110 mayfilter the set of objects to exclude those determined to be irrelevant(e.g., minor potholes, small rocks, etc.). At step 552, processing unit110 may group together the segments detected in step 550 belonging tothe same road mark or lane mark. Based on the grouping, processing unit110 may develop a model to represent the detected segments, such as amathematical model.

At step 554, processing unit 110 may construct a set of measurementsassociated with the detected segments. In some embodiments, processingunit 110 may create a projection of the detected segments from the imageplane onto the real-world plane. The projection may be characterizedusing a 3rd-degree polynomial having coefficients corresponding tophysical properties such as the position, slope, curvature, andcurvature derivative of the detected road. In generating the projection,processing unit 110 may take into account changes in the road surface,as well as pitch and roll rates associated with vehicle 200. Inaddition, processing unit 110 may model the road elevation by analyzingposition and motion cues present on the road surface. Further,processing unit 110 may estimate the pitch and roll rates associatedwith vehicle 200 by tracking a set of feature points in the one or moreimages.

At step 556, processing unit 110 may perform multi-frame analysis by,for example, tracking the detected segments across consecutive imageframes and accumulating frame-by-frame data associated with detectedsegments. As processing unit 110 performs multi-frame analysis, the setof measurements constructed at step 554 may become more reliable andassociated with an increasingly higher confidence level. Thus, byperforming steps 550-556, processing unit 110 may identify road marksappearing within the set of captured images and derive lane geometryinformation. Based on the identification and the derived information,processing unit 110 may cause one or more navigational responses invehicle 200, as described in connection with FIG. 5A, above.

At step 558, processing unit 110 may consider additional sources ofinformation to further develop a safety model for vehicle 200 in thecontext of its surroundings. Processing unit 110 may use the safetymodel to define a context in which system 100 may execute autonomouscontrol of vehicle 200 in a safe manner. To develop the safety model, insome embodiments, processing unit 110 may consider the position andmotion of other vehicles, the detected road edges and barriers, and/orgeneral road shape descriptions extracted from map data (such as datafrom map database 160). By considering additional sources ofinformation, processing unit 110 may provide redundancy for detectingroad marks and lane geometry and increase the reliability of system 100.

FIG. 5D is a flowchart showing an exemplary process 500D for detectingtraffic lights in a set of images, consistent with disclosedembodiments. Processing unit 110 may execute monocular image analysismodule 402 to implement process 500D. At step 560, processing unit 110may scan the set of images and identify objects appearing at locationsin the images likely to contain traffic lights. For example, processingunit 110 may filter the identified objects to construct a set ofcandidate objects, excluding those objects unlikely to correspond totraffic lights. The filtering may be done based on various propertiesassociated with traffic lights, such as shape, dimensions, texture,position (e.g., relative to vehicle 200), and the like. Such propertiesmay be based on multiple examples of traffic lights and traffic controlsignals and stored in a database. In some embodiments, processing unit110 may perform multi-frame analysis on the set of candidate objectsreflecting possible traffic lights. For example, processing unit 110 maytrack the candidate objects across consecutive image frames, estimatethe real-world position of the candidate objects, and filter out thoseobjects that are moving (which are unlikely to be traffic lights). Insome embodiments, processing unit 110 may perform color analysis on thecandidate objects and identify the relative position of the detectedcolors appearing inside possible traffic lights.

At step 562, processing unit 110 may analyze the geometry of a junction.The analysis may be based on any combination of: (i) the number of lanesdetected on either side of vehicle 200, (ii) markings (such as arrowmarks) detected on the road, and (iii) descriptions of the junctionextracted from map data (such as data from map database 160). Processingunit 110 may conduct the analysis using information derived fromexecution of monocular analysis module 402. In addition, Processing unit110 may determine a correspondence between the traffic lights detectedat step 560 and the lanes appearing near vehicle 200.

As vehicle 200 approaches the junction, at step 564, processing unit 110may update the confidence level associated with the analyzed junctiongeometry and the detected traffic lights. For instance, the number oftraffic lights estimated to appear at the junction as compared with thenumber actually appearing at the junction may impact the confidencelevel. Thus, based on the confidence level, processing unit 110 maydelegate control to the driver of vehicle 200 in order to improve safetyconditions. By performing steps 560-564, processing unit 110 mayidentify traffic lights appearing within the set of captured images andanalyze junction geometry information. Based on the identification andthe analysis, processing unit 110 may cause one or more navigationalresponses in vehicle 200, as described in connection with FIG. 5A,above.

FIG. 5E is a flowchart showing an exemplary process 500E for causing oneor more navigational responses in vehicle 200 based on a vehicle path,consistent with the disclosed embodiments. At step 570, processing unit110 may construct an initial vehicle path associated with vehicle 200.The vehicle path may be represented using a set of points expressed incoordinates (x, z), and the distance d, between two points in the set ofpoints may fall in the range of 1 to 5 meters. In one embodiment,processing unit 110 may construct the initial vehicle path using twopolynomials, such as left and right road polynomials. Processing unit110 may calculate the geometric midpoint between the two polynomials andoffset each point included in the resultant vehicle path by apredetermined offset (e.g., a smart lane offset), if any (an offset ofzero may correspond to travel in the middle of a lane). The offset maybe in a direction perpendicular to a segment between any two points inthe vehicle path. In another embodiment, processing unit 110 may use onepolynomial and an estimated lane width to offset each point of thevehicle path by half the estimated lane width plus a predeterminedoffset (e.g., a smart lane offset).

At step 572, processing unit 110 may update the vehicle path constructedat step 570. Processing unit 110 may reconstruct the vehicle pathconstructed at step 570 using a higher resolution, such that thedistance d_(k) between two points in the set of points representing thevehicle path is less than the distance d, described above. For example,the distance d_(k) may fall in the range of 0.1 to 0.3 meters.Processing unit 110 may reconstruct the vehicle path using a parabolicspline algorithm, which may yield a cumulative distance vector Scorresponding to the total length of the vehicle path (i.e., based onthe set of points representing the vehicle path).

At step 574, processing unit 110 may determine a look-ahead point(expressed in coordinates as (x₁, z₁)) based on the updated vehicle pathconstructed at step 572. Processing unit 110 may extract the look-aheadpoint from the cumulative distance vector S, and the look-ahead pointmay be associated with a look-ahead distance and look-ahead time. Thelook-ahead distance, which may have a lower bound ranging from 10 to 20meters, may be calculated as the product of the speed of vehicle 200 andthe look-ahead time. For example, as the speed of vehicle 200 decreases,the look-ahead distance may also decrease (e.g., until it reaches thelower bound). The look-ahead time, which may range from 0.5 to 1.5seconds, may be inversely proportional to the gain of one or morecontrol loops associated with causing a navigational response in vehicle200, such as the heading error tracking control loop. For example, thegain of the heading error tracking control loop may depend on thebandwidth of a yaw rate loop, a steering actuator loop, car lateraldynamics, and the like. Thus, the higher the gain of the heading errortracking control loop, the lower the look-ahead time.

At step 576, processing unit 110 may determine a heading error and yawrate command based on the look-ahead point determined at step 574.Processing unit 110 may determine the heading error by calculating thearctangent of the look-ahead point, e.g., arctan (x₁/z₁). Processingunit 110 may determine the yaw rate command as the product of theheading error and a high-level control gain. The high-level control gainmay be equal to: (2/look-ahead time), if the look-ahead distance is notat the lower bound. Otherwise, the high-level control gain may be equalto: (2*speed of vehicle 200/look-ahead distance).

FIG. 5F is a flowchart showing an exemplary process 500F for determiningwhether a leading vehicle is changing lanes, consistent with thedisclosed embodiments. At step 580, processing unit 110 may determinenavigation information associated with a leading vehicle (e.g., avehicle traveling ahead of vehicle 200). For example, processing unit110 may determine the position, velocity (e.g., direction and speed),and/or acceleration of the leading vehicle, using the techniquesdescribed in connection with FIGS. 5A and 5B, above. Processing unit 110may also determine one or more road polynomials, a look-ahead point(associated with vehicle 200), and/or a snail trail (e.g., a set ofpoints describing a path taken by the leading vehicle), using thetechniques described in connection with FIG. 5E, above.

At step 582, processing unit 110 may analyze the navigation informationdetermined at step 580. In one embodiment, processing unit 110 maycalculate the distance between a snail trail and a road polynomial(e.g., along the trail). If the variance of this distance along thetrail exceeds a predetermined threshold (for example, 0.1 to 0.2 meterson a straight road, 0.3 to 0.4 meters on a moderately curvy road, and0.5 to 0.6 meters on a road with sharp curves), processing unit 110 maydetermine that the leading vehicle is likely changing lanes. In the casewhere multiple vehicles are detected traveling ahead of vehicle 200,processing unit 110 may compare the snail trails associated with eachvehicle. Based on the comparison, processing unit 110 may determine thata vehicle whose snail trail does not match with the snail trails of theother vehicles is likely changing lanes. Processing unit 110 mayadditionally compare the curvature of the snail trail (associated withthe leading vehicle) with the expected curvature of the road segment inwhich the leading vehicle is traveling. The expected curvature may beextracted from map data (e.g., data from map database 160), from roadpolynomials, from other vehicles' snail trails, from prior knowledgeabout the road, and the like. If the difference in curvature of thesnail trail and the expected curvature of the road segment exceeds apredetermined threshold, processing unit 110 may determine that theleading vehicle is likely changing lanes.

In another embodiment, processing unit 110 may compare the leadingvehicle's instantaneous position with the look-ahead point (associatedwith vehicle 200) over a specific period of time (e.g., 0.5 to 1.5seconds). If the distance between the leading vehicle's instantaneousposition and the look-ahead point varies during the specific period oftime, and the cumulative sum of variation exceeds a predeterminedthreshold (for example, 0.3 to 0.4 meters on a straight road, 0.7 to 0.8meters on a moderately curvy road, and 1.3 to 1.7 meters on a road withsharp curves), processing unit 110 may determine that the leadingvehicle is likely changing lanes. In another embodiment, processing unit110 may analyze the geometry of the snail trail by comparing the lateraldistance traveled along the trail with the expected curvature of thesnail trail. The expected radius of curvature may be determinedaccording to the calculation: (δ_(z) ²+δ_(x) ²)/2/(δ_(x)), where δ_(x)represents the lateral distance traveled and δ_(z) represents thelongitudinal distance traveled. If the difference between the lateraldistance traveled and the expected curvature exceeds a predeterminedthreshold (e.g., 500 to 700 meters), processing unit 110 may determinethat the leading vehicle is likely changing lanes. In anotherembodiment, processing unit 110 may analyze the position of the leadingvehicle. If the position of the leading vehicle obscures a roadpolynomial (e.g., the leading vehicle is overlaid on top of the roadpolynomial), then processing unit 110 may determine that the leadingvehicle is likely changing lanes. In the case where the position of theleading vehicle is such that, another vehicle is detected ahead of theleading vehicle and the snail trails of the two vehicles are notparallel, processing unit 110 may determine that the (closer) leadingvehicle is likely changing lanes.

At step 584, processing unit 110 may determine whether or not leadingvehicle 200 is changing lanes based on the analysis performed at step582. For example, processing unit 110 may make the determination basedon a weighted average of the individual analyses performed at step 582.Under such a scheme, for example, a decision by processing unit 110 thatthe leading vehicle is likely changing lanes based on a particular typeof analysis may be assigned a value of “1” (and “0” to represent adetermination that the leading vehicle is not likely changing lanes).Different analyses performed at step 582 may be assigned differentweights, and the disclosed embodiments are not limited to any particularcombination of analyses and weights.

FIG. 6 is a flowchart showing an exemplary process 600 for causing oneor more navigational responses based on stereo image analysis,consistent with disclosed embodiments. At step 610, processing unit 110may receive a first and second plurality of images via data interface128. For example, cameras included in image acquisition unit 120 (suchas image capture devices 122 and 124 having fields of view 202 and 204)may capture a first and second plurality of images of an area forward ofvehicle 200 and transmit them over a digital connection (e.g., USB,wireless, Bluetooth, etc.) to processing unit 110. In some embodiments,processing unit 110 may receive the first and second plurality of imagesvia two or more data interfaces. The disclosed embodiments are notlimited to any particular data interface configurations or protocols.

At step 620, processing unit 110 may execute stereo image analysismodule 404 to perform stereo image analysis of the first and secondplurality of images to create a 3D map of the road in front of thevehicle and detect features within the images, such as lane markings,vehicles, pedestrians, road signs, highway exit ramps, traffic lights,road hazards, and the like. Stereo image analysis may be performed in amanner similar to the steps described in connection with FIGS. 5A-5D,above. For example, processing unit 110 may execute stereo imageanalysis module 404 to detect candidate objects (e.g., vehicles,pedestrians, road marks, traffic lights, road hazards, etc.) within thefirst and second plurality of images, filter out a subset of thecandidate objects based on various criteria, and perform multi-frameanalysis, construct measurements, and determine a confidence level forthe remaining candidate objects. In performing the steps above,processing unit 110 may consider information from both the first andsecond plurality of images, rather than information from one set ofimages alone. For example, processing unit 110 may analyze thedifferences in pixel-level data (or other data subsets from among thetwo streams of captured images) for a candidate object appearing in boththe first and second plurality of images. As another example, processingunit 110 may estimate a position and/or velocity of a candidate object(e.g., relative to vehicle 200) by observing that the object appears inone of the plurality of images but not the other or relative to otherdifferences that may exist relative to objects appearing in the twoimage streams. For example, position, velocity, and/or accelerationrelative to vehicle 200 may be determined based on trajectories,positions, movement characteristics, etc., of features associated withan object appearing in one or both of the image streams.

At step 630, processing unit 110 may execute navigational responsemodule 408 to cause one or more navigational responses in vehicle 200based on the analysis performed at step 620 and the techniques asdescribed above in connection with FIG. 4. Navigational responses mayinclude, for example, a turn, a lane shift, a change in acceleration, achange in velocity, braking, and the like. In some embodiments,processing unit 110 may use data derived from execution of velocity andacceleration module 406 to cause the one or more navigational responses.Additionally, multiple navigational responses may occur simultaneously,in sequence, or any combination thereof.

FIG. 7 is a flowchart showing an exemplary process 700 for causing oneor more navigational responses based on an analysis of three sets ofimages, consistent with disclosed embodiments. At step 710, processingunit 110 may receive a first, second, and third plurality of images viadata interface 128. For instance, cameras included in image acquisitionunit 120 (such as image capture devices 122, 124, and 126 having fieldsof view 202, 204, and 206) may capture a first, second, and thirdplurality of images of an area forward and/or to the side of vehicle 200and transmit them over a digital connection (e.g., USB, wireless,Bluetooth, etc.) to processing unit 110. In some embodiments, processingunit 110 may receive the first, second, and third plurality of imagesvia three or more data interfaces. For example, each of image capturedevices 122, 124, 126 may have an associated data interface forcommunicating data to processing unit 110. The disclosed embodiments arenot limited to any particular data interface configurations orprotocols.

At step 720, processing unit 110 may analyze the first, second, andthird plurality of images to detect features within the images, such aslane markings, vehicles, pedestrians, road signs, highway exit ramps,traffic lights, road hazards, and the like. The analysis may beperformed in a manner similar to the steps described in connection withFIGS. 5A-5D and 6, above. For instance, processing unit 110 may performmonocular image analysis (e.g., via execution of monocular imageanalysis module 402 and based on the steps described in connection withFIGS. 5A-5D, above) on each of the first, second, and third plurality ofimages. Alternatively, processing unit 110 may perform stereo imageanalysis (e.g., via execution of stereo image analysis module 404 andbased on the steps described in connection with FIG. 6, above) on thefirst and second plurality of images, the second and third plurality ofimages, and/or the first and third plurality of images. The processedinformation corresponding to the analysis of the first, second, and/orthird plurality of images may be combined. In some embodiments,processing unit 110 may perform a combination of monocular and stereoimage analyses. For example, processing unit 110 may perform monocularimage analysis (e.g., via execution of monocular image analysis module402) on the first plurality of images and stereo image analysis (e.g.,via execution of stereo image analysis module 404) on the second andthird plurality of images. The configuration of image capture devices122, 124, and 126—including their respective locations and fields ofview 202, 204, and 206—may influence the types of analyses conducted onthe first, second, and third plurality of images. The disclosedembodiments are not limited to a particular configuration of imagecapture devices 122, 124, and 126, or the types of analyses conducted onthe first, second, and third plurality of images.

In some embodiments, processing unit 110 may perform testing on system100 based on the images acquired and analyzed at steps 710 and 720. Suchtesting may provide an indicator of the overall performance of system100 for certain configurations of image capture devices 122, 124, and126. For example, processing unit 110 may determine the proportion of“false hits” (e.g., cases where system 100 incorrectly determined thepresence of a vehicle or pedestrian) and “misses.”

At step 730, processing unit 110 may cause one or more navigationalresponses in vehicle 200 based on information derived from two of thefirst, second, and third plurality of images. Selection of two of thefirst, second, and third plurality of images may depend on variousfactors, such as, for example, the number, types, and sizes of objectsdetected in each of the plurality of images. Processing unit 110 mayalso make the selection based on image quality and resolution, theeffective field of view reflected in the images, the number of capturedframes, the extent to which one or more objects of interest actuallyappear in the frames (e.g., the percentage of frames in which an objectappears, the proportion of the object that appears in each such frame,etc.), and the like.

In some embodiments, processing unit 110 may select information derivedfrom two of the first, second, and third plurality of images bydetermining the extent to which information derived from one imagesource is consistent with information derived from other image sources.For example, processing unit 110 may combine the processed informationderived from each of image capture devices 122, 124, and 126 (whether bymonocular analysis, stereo analysis, or any combination of the two) anddetermine visual indicators (e.g., lane markings, a detected vehicle andits location and/or path, a detected traffic light, etc.) that areconsistent across the images captured from each of image capture devices122, 124, and 126. Processing unit 110 may also exclude information thatis inconsistent across the captured images (e.g., a vehicle changinglanes, a lane model indicating a vehicle that is too close to vehicle200, etc.). Thus, processing unit 110 may select information derivedfrom two of the first, second, and third plurality of images based onthe determinations of consistent and inconsistent information.

Navigational responses may include, for example, a turn, a lane shift, achange in acceleration, and the like. Processing unit 110 may cause theone or more navigational responses based on the analysis performed atstep 720 and the techniques as described above in connection with FIG.4. Processing unit 110 may also use data derived from execution ofvelocity and acceleration module 406 to cause the one or morenavigational responses. In some embodiments, processing unit 110 maycause the one or more navigational responses based on a relativeposition, relative velocity, and/or relative acceleration betweenvehicle 200 and an object detected within any of the first, second, andthird plurality of images. Multiple navigational responses may occursimultaneously, in sequence, or any combination thereof.

Predicting Cut In Vehicles and Altruistic Behavioral Responses

During navigation, an autonomous vehicle, such as vehicle 200, mayencounter another vehicle that is attempting a lane shift. For example,a vehicle in a lane (e.g., a lane designated by markings on a roadway ora lane aligned with the path of vehicle 200 without markings on theroad) to the left or to the right of the lane in which vehicle 200 istraveling may attempt to shift, or cut in, to the lane in which vehicle200 is traveling. Such a vehicle may be referred to as a target vehicle.When such a cut in occurs, vehicle 200 may need to make a navigationalresponse. For example, vehicle 200 could change its velocity oracceleration and/or shift to another lane to avoid the cut-in by thetarget vehicle.

In some instances, the target vehicle may appear to attempt a cut in,but the cut in may ultimately not be completed. A driver of the targetvehicle (or even a fully or partially autonomous navigational systemassociated with the target vehicle) may, for example, change his or hermind or otherwise change a navigational plan away from a lane change, orthe target vehicle may simply have been drifting. Accordingly, in orderto avoid frequent unnecessary braking and/or accelerations, it may bedesirable for vehicle 200 to delay effecting a navigational responseuntil a cut in by the target vehicle is determined to be sufficientlylikely. On the other hand, in some situations (especially where a changein course of the target vehicle into the path of the host vehicle isexpected), it may be desirable for vehicle 200 to effect a navigationalresponse earlier. Such navigation based at least in part on expectedbehavior may help avoid sudden braking and may provide an even furtherincreased safety margin. Improved prediction of when the target vehiclewill attempt a cut in can help minimize both unnecessary braking andsudden braking. Such an improved prediction may be referred to as cut indetection, and the navigational responses taken when a cut in isdetected may be referred to as a cut in response.

In some embodiments, such improved prediction may rely on monocularand/or stereo image analysis and/or information obtained from othersources (e.g., from a GPS device, a speed sensor, an accelerometer, asuspension sensor, etc.) to detect, for example, static road features(e.g., a lane ending, a roadway split), dynamic road features (e.g., thepresence of other vehicles ahead of the vehicle likely to attempt a cutin), and/or traffic rules and driving customs in a geographic area.These static road features, dynamic road features, and/or traffic rulesand driving customs may be referred to as predetermined cut insensitivity change factors, and the presence of one or morepredetermined cut in sensitivity change factors may cause vehicle 200 tomodify its sensitivity to an attempted cut in by the target vehicle. Forexample, where a predetermined cut in sensitivity change factor ispresent in an environment (e.g., the target vehicle is closely trailinganother vehicle moving at a lower speed), vehicle 200 may rely on a afirst cut in sensitivity parameter, and where no predetermined cut insensitivity change factor is present in the environment (e.g., thetarget vehicle is the only vehicle in its lane), vehicle 200 may rely ona second cut in sensitivity parameter. The second cut in sensitivityparameter may be different than (e.g., more sensitive than) the firstcut in sensitivity parameter.

In some cases, a cut in by the target vehicle may be necessary. Forexample, the lane in which the target vehicle is traveling may beending, the roadway may be splitting, or there may be an obstacle (e.g.,a stopped vehicle, an object, or other type of blockage) in the lane inwhich the target vehicle is traveling. In other cases, though, a cut inby the target vehicle may be optional. For example, the target vehiclemay attempt a cut in merely to pass a slower moving vehicle. When a cutin is optional, whether the target vehicle attempts the cut in maydepend on how vehicle 200 (e.g., a host vehicle) behaves. For example,vehicle 200 may decelerate to signal that the target vehicle may cut inor may accelerate to signal that the target vehicle may not cut in.While accelerating may, in many cases, be desirable for vehicle 200,insofar as it allows vehicle 200 to reach a destination more quickly, anoperator of vehicle 200 (or navigational system in full or partialcontrol of vehicle 200) may be inclined to allow the cut in in some orall cases. This may be referred to as altruistic behavior. Thus, it maybe desirable for vehicle 200 to take into account altruistic behaviorconsiderations in determining whether to permit a cut in by a targetvehicle.

FIG. 8 is another exemplary functional block diagram of memory 140and/or 150, which may be stored/programmed with instructions forperforming one or more operations consistent with the disclosedembodiments. Although the following refers to memory 140, one of skillin the art will recognize that instructions may be stored in memory 140and/or 150, and the instructions may be executed by processing unit 110of system 100.

As shown in FIG. 8, memory 140 may store monocular image analysis module402, stereo image analysis module 404, velocity and acceleration module406, and navigational response module 408, which may take any of theforms of the modules described above in connection with FIG. 4. Memory140 may further store a cut in detection module 802, a cut in responsemodule 804, and an altruistic behavior module 806. The disclosedembodiments are not limited to any particular configuration of memory140. Further, applications processor 180 and/or image processor 190 mayexecute the instructions stored in any of modules 402-408 and 802-806included in memory 140. One of skill in the art will understand thatreferences in the following discussions to processing unit 110 may referto applications processor 180 and image processor 190 individually orcollectively. Accordingly, steps of any of the following processes maybe performed by one or more processing devices. Further, any of modules402-408 and 802-806 may be stored remotely from vehicle 200 (e.g.,distributed over one or more servers in communication with a network andaccessible over the network via wireless transceiver 172 of vehicle200).

In some embodiments, cut in detection module 802 may store instructionsthat, when executed by processing unit 110, enable detection of a targetvehicle. The target vehicle may be a vehicle traveling in a laneadjacent to the lane in which vehicle 200 is traveling and, in somecases, may be a leading vehicle. In some embodiments, cut in detectionmodule 802 may detect the target vehicle by performing monocular imageanalysis of a set of images acquired by one of image capture devices122, 124, and 126, as described above in connection with monocular imageanalysis module 402. In some embodiments, processing unit 110 maycombine information from a set of images with additional sensoryinformation (e.g., information from radar or lidar) to perform themonocular image analysis. As described in connection with FIGS. 5A-5Dabove, such monocular image analysis may involve detecting a set offeatures within the set of images, such as vehicle edge features,vehicle lights (or other elements associated with the vehicle), lanemarkings, vehicles, or, road signs. For example, detecting the targetvehicle may be carried out as described in connection with FIG. 5B,including determining a set of candidate objects that includes thetarget vehicle, filtering the set of candidate objects, performingmulti-frame analysis of the set of candidate objects, constructing a setof measurements for the detected objects (including the target vehicle),and performing optical flow analysis. The measurements may include, forexample, position, velocity, and acceleration values (relative tovehicle 200) associated with the target vehicle.

Alternatively or additionally, in some embodiments cut in detectionmodule 802 may detect the target vehicle by performing stereo imageanalysis of first and second sets of images acquired by a combination ofimage capture devices selected from any of image capture devices 122,124, and 126, as described above in connection with stereo imageanalysis module 404. In some embodiments, processing unit 110 maycombine information from the first and second sets of images withadditional sensory information (e.g., information from radar or lidar)to perform the stereo image analysis. For example, stereo image analysismodule 404 may include instructions for performing stereo image analysisbased on a first set of images acquired by image capture device 124 anda second set of images acquired by image capture device 126. Asdescribed in connection with FIG. 6 above, stereo image analysis mayinvolve detecting a set of features within the first and second sets ofimages, such as vehicle edge features, vehicle lights (or other elementsassociated with the vehicle), lane markings, vehicles, or, road signs.

In other embodiments, as an alternative to detecting one or morevehicles (e.g., the target vehicle) by analyzing images acquired by oneof image capture devices 122, 124, and 126, cut in detection module 802may instead detect a vehicle through analysis of sensory information,such as information acquired via a radar device or lidar device includedin system 100.

In some embodiments, cut in detection module 802 may further storeinstructions that, when executed by processing unit 110, enableidentification of an indicator that the target vehicle will attempt acut in (that is, that the target vehicle will attempt to change lanesinto the lane in which vehicle 200 is traveling or otherwise move into atravel path of the host vehicle). In some embodiments, identifying theindicator may involve using monocular and/or stereo image analysis todetect a position and/or speed of the target vehicle, as described abovein connection with FIG. 5B. In some embodiments, identifying theindicator may further involve detecting one or more road markings, asdescribed above in connection with FIG. 5C. And in some embodimentsidentifying the indicator may further involve detecting that the targetvehicle is changing lanes, as described above in connection with FIG.5F. In particular, processing unit 110 may determine navigationinformation associated with the target vehicle, such as position,velocity (e.g., direction and speed), and/or acceleration of the leadingvehicle, using the techniques described in connection with FIGS. 5A and5B, above. Processing unit 110 may also determine one or more roadpolynomials, a look-ahead point (associated with vehicle 200), and/or asnail trail (e.g., a set of points describing a path taken by theleading vehicle), using the techniques described in connection with FIG.5E, above. Further, processing unit 110 may analyze the navigationinformation by, for example, calculating the distance between a snailtrail and a road polynomial (e.g., along the trail) or comparing theleading vehicle's instantaneous position with the look-ahead point(associated with vehicle 200) over a specific period of time (e.g., 0.5to 1.5 seconds), as described above in connection with FIG. 5F. Based onthe analysis, the processing unit 110 may identify whether an indicatorthat the leading vehicle is attempting a cut in is present.

Cut in detection module 802 may further store instructions that, whenexecuted by processing unit 110, enable detection that a predeterminedcut in sensitivity change factor is present in the environment ofvehicle 200. A predetermined cut in sensitivity change factor mayinclude any indicator suggestive of a tendency of the target vehicle toremain on a current course or to change course into a path of the hostvehicle. Such sensitivity change factors may include static roadfeatures (e.g., a lane ending, a roadway split, a barrier, an object),dynamic road features (e.g., the presence of other vehicles ahead of thevehicle likely to attempt a cut in), and/or traffic rules and drivingcustoms in a geographic area. In some embodiments, detecting apredetermined cut in sensitivity change factor may involve usingmonocular and/or stereo image analysis to detect set of features withinthe set of images, such as lane markings, vehicles, pedestrians, roadsigns, highway exit ramps, traffic lights, hazardous objects, and anyother feature associated with an environment of a vehicle as describedabove in connection with FIGS. 5A-5D and 6. Alternatively oradditionally, in some embodiments detecting a predetermined cut insensitivity change factor may involve using other sensory information,such as GPS data. For example, a lane ending, highway split, etc. may bedetermined based on map data. As another example, a traffic rule may bedetermined based on GPS location data.

Cut in response module 804 may store instructions that, when executed byprocessing unit 110, enable a cut in response to be effected. A cut inresponse may take any of the forms described above for a navigationalresponse, such as a turn, a lane shift, a change in acceleration, andthe like, as discussed below in connection with navigational responsemodule 408.

In some embodiments, whether a cut in response is effected may depend onthe presence of absence of a predetermined cut in sensitivity changefactor in the environment. For example, where no predetermined cut insensitivity change factor is present in an environment (e.g., thevehicle attempting a cut in is the only vehicle in its lane, no laneshifts or obstacles are detected either through image analysis or reviewof GPS/map/traffic data, etc.), vehicle 200 may rely on a first cut insensitivity parameter. In some embodiments, where no predetermined cutin sensitivity change factor is present in an environment, vehicle 200may rely on a value associated with the first cut in sensitivityparameter. The value may include any value directly tied to and/ormeasured with respect to the first cut in sensitivity parameter or maybe one or more values indirectly related to the first cut in sensitivityparameter.

On the other hand, where one or more predetermined cut in sensitivitychange factors are present in the environment (e.g., the target vehicleis moving at a high speed and is approaching another vehicle in its laneat a lower speed, a lane shift or lane end condition is detected eithervisually through image analysis or through reference to GPS/map/trafficinformation, the local region is one that discourages left lane driving,etc.), vehicle 200 may rely on a second cut in sensitivity parameter.The second cut in sensitivity parameter may be different than (e.g.,more sensitive than) the first cut in sensitivity parameter. In someembodiments, where one or more predetermined cut in sensitivity changefactors are present in the environment, vehicle 200 may rely on a valueassociated with the second cut in sensitivity parameter. Similar to thediscussion above regarding the value associated with the first cut insensitivity parameter, the value associated with the second cut insensitive parameter may include any value directly tied to and/ormeasured with respect to the second cut in sensitivity parameter or maybe one or more values indirectly related to the second cut insensitivity parameter.

The first and second cut in sensitivity parameters may establish twostates of operation: a first in which the host vehicle may be lesssensitive to movements or course changes by a target vehicle, and asecond state in which the host vehicle may be more sensitive tomovements or course changes by a target vehicle. Such a state approachcan reduce the number or degree of decelerations, accelerations, orcourse changes in the first state when a course change by the targetvehicle is not expected based on any environmental conditions. In thesecond state, however, where at least one condition is detected ordetermined in the environment of the target vehicle that is expected toresult in a course change (or other navigational response) of the targetvehicle, the host vehicle can be more sensitive to course changes ornavigational changes of the target vehicle and may alter course,accelerate, slow, etc. based on even small changes recognized in thenavigation of the target vehicle. Such small changes, in the secondstate, may be interpreted as consistent with an expected navigationalchange by the target vehicle, and therefore, may warrant an earlierresponse, as compared to navigation within the first state.

It is further possible, that in the second state, course changes of thehost vehicle may be made based on detection of the sensitivity changefactor alone and without detection of a navigational response or changein the target vehicle. For example, where image analysis in the hostvehicle system indicates that a sensitivity change factor exists (e.g.,by recognizing in the captured images a road sign indicative of a laneend condition, recognizing in the images a lane end ahead of the targetvehicle, recognizing an object ahead of the target vehicle or a slowermoving vehicle, or any other sensitivity change factor), the hostvehicle may take one or more preemptive actions (slow, accelerate,change course, etc.) even without detecting a navigational change by thetarget vehicle. Such a change may be warranted based on the expectationthat the target vehicle will need to make a navigational change in thefuture. Such a change may increase or maximize an amount of time betweenthe host vehicle navigational change and the expected target vehiclenavigational change.

A cut in sensitivity parameter may take the form of a threshold (e.g., athreshold distance between a snail trail and a road polynomial (e.g.,along the trail) or between the target vehicle's instantaneous positionwith a look-ahead point (associated with vehicle 200) over a specificperiod of time (e.g., 0.5 to 1.5 seconds), a combination of suchthresholds (e.g., a threshold lateral speed of the target vehicle incombination with a threshold lateral position of the vehicle relative tovehicle 200), and/or a weighted average of such thresholds, as describedabove in connection with FIG. 5F. As discussed above, in someembodiments, the cut in sensitivity parameter may also be anyarbitrarily assigned variable for which a value is associated with theparameter or any value indirectly related to the value of the parametermay be used to create the two (or more) sensitivity states describedabove. For example, such a variable may be associated with a first valueindicative of a low sensitivity state where no sensitivity change factoris detected, and may be associated with a second value indicative of ahigher sensitivity state where a sensitivity change factor is found inthe environment of the target vehicle.

In some embodiments, cut in response module 804 may evaluate a distancefrom vehicle 200 to a vehicle traveling behind vehicle 200 and factorthat distance into a cut sensitivity parameter. For example, vehicle 200may include one or more rear facing sensors (e.g., a radar sensor) todetermine a distance from vehicle 200 to a vehicle traveling behindvehicle 200. Still further, in some embodiments, vehicle 200 may includeone more rear facing cameras that provide images to system 100 foranalysis to determine a distance from vehicle 200 to a vehicle travelingbehind vehicle 200. In still other embodiments, system 100 may use datafrom one or more sensors and one or more rear facing cameras todetermine a distance from vehicle 200 to a vehicle traveling behindvehicle 200. Based on the distance from vehicle 200, cut in responsemodule 804 may evaluate whether or not decelerating is safe for vehicle200 based on the distance between vehicle 200 and the vehicle travelingbehind vehicle 200.

Alternatively or additionally, the cut in sensitivity parameters may bederived from examples using machine learning techniques such as neuralnetworks. For example, a neural network may be trained to determine cutin sensitivity parameters based on scenarios. As an example, the neuralnetwork could be trained for a scenario in which a roadway includesthree lanes of travel, vehicle 200 is in the center lane, there is aleading vehicle in the center lane, two vehicles are in the left lane,and one vehicle is in the right lane. The scenario could specify to theneural network a longitudinal position, longitudinal speed, lateralposition, and lateral speed of each of the vehicles. Based on thescenario, the neural network may determine, for instance, a binaryoutput (e.g., whether a target vehicle will attempt a cut in in the nextN frames) or a value output (e.g., how long until the target vehicleattempts to cut in to the center lane). The binary or value output, oranother value derived from these outputs, may be used as the cut insensitivity parameter. The neural network may be similarly trained forother scenarios, such as scenarios in which the roadway includes onlytwo lanes of travel, there is no leading vehicle in the center lane, orthere is no vehicle in the right lane. In some embodiments, the neuralnetwork may be provided via one or more programming modules stored inmemory 140 and/or 150. In still embodiments, in addition to or as analternative to memory 140 and/or 150, the neural network (or aspects ofthe neural network) may be provided via one or more servers locatedremotely from vehicle 200 and accessible over a network via wirelesstransceiver 172.

In some embodiments, to minimize the number of scenarios, the scenariosmay be grouped according to the number of lanes in the roadway and thelocation of vehicle 200. For example, the scenarios may include ascenario having two lanes in which vehicle 200 is in the left lane, ascenario having two lanes in which vehicle 200 is in the right lane, ascenario having three lanes in which vehicle 200 is in the left lane, ascenario having three lanes for which vehicle 200 is in the center lane,a scenario having three lanes for which vehicle 200 is in the rightlane, a scenario having four lanes in which vehicle 200 is in theleftmost lane, a scenario having four lanes in which vehicle 200 is inthe center left lane, etc. In each scenario, a leading vehicle may betraveling in the same lane as vehicle 200 and two vehicles may betraveling in each of the other lanes. To account for scenarios in whichone or more of these other vehicles is absent (e.g., there is no leadingvehicle, there is only one vehicle in the right lane, etc.), thelongitudinal distance of the absent vehicle may be set to infinity.

Once vehicle 200 has detected the target vehicle (e.g., has identifiedan indicator that the target vehicle is attempting a cut in), anddetected whether any cut in sensitivity change factor is present in theenvironment, as described above, cut in response module 804 may providethe target vehicle, indicator, and/or (if present) the sensitivitychange factor(s) to the neural network, and the neural network mayselect the scenario that most closely resembles the situation of vehicle200.

Based on the selected scenario, the neural network may indicate a binaryoutput (e.g., whether a target vehicle will attempt a cut in in the nextN frames) or a value output (e.g., how long until the target vehicleattempts to cut in to the center lane), as described above, from which acut in sensitivity parameter may be derived. Based on the cut insensitivity parameter, cut in response module 804 enable a cut inresponse to be effected.

As noted above, a cut in response may take any of the forms describedabove for a navigational response, such as a turn, a lane shift, and/ora change in acceleration, and the like, as discussed above in connectionwith navigational response module 408. In some embodiments, processingunit 110 may use data derived from execution of velocity andacceleration module 406, described above, to cause the one or more cutin responses. Additionally, multiple cut in responses may occursimultaneously, in sequence, or any combination thereof. For instance,processing unit 110 may cause vehicle 200 to shift one lane over andthen accelerate by, for example, sequentially transmitting controlsignals to steering system 240 and throttling system 220 of vehicle 200.Alternatively, processing unit 110 may cause vehicle 200 to brake whileat the same time shifting lanes by, for example, simultaneouslytransmitting control signals to braking system 230 and steering system240 of vehicle 200. In some embodiments, cut in response module 804 mayanalyze images acquired by one of image capture devices 122, 124, and126 to detect whether any cut in sensitivity change factor is present inthe environment of vehicle 200. In other embodiments, in addition to oras an alternative to analyzing images, cut in response module 804 maydetect whether any cut in sensitivity change factor is present in theenvironment of vehicle 200 through analysis of sensory information, suchas information acquired via a radar device or lidar device included insystem 100.

In some embodiments, the neural networks described above may be furtherconfigured to determine the cut in response as well. For example, theneural networks may determine and output, based on the detected targetvehicle, indicator, and/or (if present) the sensitivity changefactor(s), indicating what navigational response should be effected. Insome embodiments, such an output may be determined based on previouslydetected driver behavior. In some embodiments, previously detecteddriver behavior may be filtered using a cost function analysis thatpreferences smooth driving behavior (e.g., smooth driving behavior canbe measured as that which minimizes the square of the deceleration oracceleration integration over a period of time).

Cut in detection module 802 and cut in response module 804 are furtherdescribed below in connection with FIGS. 9A-9E and 11.

Altruistic behavior module 806 may store instructions that, whenexecuted by processing unit 110, taken into account altruistic behaviorconsiderations to determine whether vehicle 200 should permit a targetvehicle to cut into the lane in which vehicle 200 is traveling. To thisend, processing unit 110 may detect the target vehicle in any of themanners described above in connection with cut in detection module 802.

Further, processing unit 110 may determine one or more situationalcharacteristics associated with the target vehicle. A situationalcharacteristic may be, for example, any characteristic that indicatesthe target vehicle would benefit from changing lanes into the lane inwhich vehicle 200 is traveling. For example, a situationalcharacteristic may indicate that the target vehicle is traveling in alane adjacent to the lane in which vehicle 200 is traveling and that thetarget vehicle is behind another vehicle that is traveling more slowlythan the target vehicle. In general, the situational characteristic mayindicate that, while a cut in or other navigational response by thetarget vehicle may not be necessary, a cut in would benefit the targetvehicle. Other such situations may occur, for example, in traffic jams,at crowded roundabouts, at lane end situations, or any other conditionswhere a target vehicle may desire to move into a path of a host vehicle.

In some embodiments, detecting the situational characteristic(s) mayinvolve using monocular and/or stereo image analysis to detect a set offeatures within a set of images, such as lane markings, vehicles,pedestrians, road signs, highway exit ramps, traffic lights, hazardousobjects, and any other feature associated with an environment of avehicle as described above in connection with FIGS. 5A-5D and 6. Forexample, identifying the situational characteristic(s) may involve usingmonocular and/or stereo image analysis to detect a position and/or speedof the target vehicle and/or one or more other vehicles, as describedabove in connection with FIG. 5B. In some embodiments, identifying thesituational characteristic(s) may further involve detecting one or moreroad markings, as described above in connection with FIG. 5C.Alternatively or additionally, in some embodiments detecting thesituational characteristic(s) may involve using other sensoryinformation, such as GPS data. For example, an entrance ramp in theroadway may be determined based on map data. In some situations,detecting the situational characteristic may involve determining fromcaptured images a location of a target vehicle relative to othervehicles and/or a number of vehicles in a vicinity of the targetvehicle, etc. In still other embodiments, detecting the situationalcharacteristic(s) may involve using other sensory information, such asinformation acquired via a radar device or lidar device included insystem 100.

Altruistic behavior module 806 may further store instructions that, whenexecuted by processing unit 110, determine whether to cause anavigational change (e.g., any of the navigational responses describedabove) that would permit the target vehicle to cut into the lane inwhich vehicle 200 is traveling. Such a determination may be made basedon an altruistic behavior parameter.

The altruistic behavior parameter may, for example, be set based oninput from an operator of vehicle 200. For instance, the operator mayset the altruistic behavior parameter to allow all target vehicles tocut into the lane in which vehicle 200 is traveling, to allow only oneof every n target vehicles to cut in to the lane in which vehicle 200 istraveling, to allow a target vehicle to cut into the lane in whichvehicle 200 is traveling only when another vehicle ahead of the targetvehicle is traveling below a certain speed, etc. Alternatively oradditionally, in some embodiments the altruistic behavior parameter maybe user-selectable (e.g., a user-selectable value or state) before orduring navigation. For example, a user may select (e.g., through userinterface 170), upon entering vehicle 200, whether to be altruistic ornot during navigation. As another example, the user may select (e.g.,through user interface 170), when one or more situationalcharacteristics are detected, whether to be altruistic or not in thatinstance.

In some embodiments, the altruistic behavior parameter may be set basedon at least one informational element determined by parsing calendarentries for an operator or a passenger of vehicle 200. For example, insome embodiments the altruistic behavior parameter may indicate that atarget vehicle should be allowed to cut into the lane in which vehicle200 is traveling so long as vehicle 200 will reach a destination withina time frame that is acceptable to an operator of vehicle 200. Forinstance, when the operator or a passenger is not in a hurry, thecriteria may allow more target vehicles to cut in, but when the operatoror passenger is in a hurry, the criteria may allow fewer target vehiclesto cut in or may not allow any target vehicles to cut in at all. Whetheran operator or passenger is in a hurry may be determined by theprocessing unit 110 based on, for instance, calendar entries associatedwith the operator or passenger (e.g., a calendar event indicating thatthe user wishes to arrive at a certain location by a certain time) andnavigational data for vehicle 200 (e.g., GPS and/or map data estimatingan arrival time at the location).

In some embodiments, the altruistic behavior parameter may be set basedon an output of a randomizer function. For instance, certain outputs ofthe randomizer function may cause vehicle 200 to allow the targetvehicle 1002 to cut in to the first lane 1004, while other outputs ofthe randomizer function may cause vehicle 200 to not allow the targetvehicle 1002 to cut into the first lane 1004. Still alternatively oradditionally, the altruistic behavior parameter may be set based on adetermined number of encounters with target vehicles for which the oneor more situational characteristics indicate that the target vehiclewould benefit from a course change into the first lane 1004.

In some embodiments, the altruistic behavior parameter may be fixed.Alternatively, the altruistic behavior parameter may be updated suchthat a navigational change in vehicle 200 is caused in at least apredetermined percentage of encounters with target vehicles for whichthe one or more situational characteristics indicate that the targetvehicle would benefit from a course change into a path of vehicle 200.For example, the predetermined percentage may be, e.g., at least 10%, atleast 20%, at least 30%, etc.

In some embodiments, the altruistic behavior parameter may specifyrules, such that a navigational change should be effected if certaincriteria are met. For instance, a predetermined altruistic behaviorparameter may indicate that, if vehicle 200 is approaching a targetvehicle in an adjacent lane and the target vehicle is traveling behindanother vehicle that is moving more slowly than the target vehicle,vehicle 200 should decelerate to permit the target vehicle to cut in solong as: the target vehicle has indicated a desire to cut in (e.g.,through the use of a blinker or through lateral movement towards thelane in which vehicle 200 is traveling), the deceleration of vehicle 200is below a certain threshold (e.g., to avoid braking too rapidly), andthere is no vehicle behind vehicle 200 that would make decelerationunsafe.

In some embodiments, the altruistic behavior parameter may be designedto be inconsistent, such that the same situational characteristic(s) andcriteria may result in different navigational changes or no navigationalchanges at all. For instance, the altruistic behavior parameter mayresult in altruism variations that are random and/or cyclical. As anexample, the altruistic behavior parameter may permit only one in everyn target vehicles to cut in. In some embodiments, n may be randomlyselected, may vary randomly, may increase each time situationalcharacteristic(s) are detected, and/or may be reset to a low value whena cut in is permitted. As another example, the altruistic behaviorparameter may determine whether or not to permit the target vehicle tocut in based on comparison of a randomly generated number to apredetermined threshold.

In some embodiments, altruistic behavior module 804 may consider howmany vehicles are traveling behind vehicle 200 and factor thatinformation into the altruistic behavior parameter. For example, vehicle200 may include one or more rear facing sensors (e.g., a radar sensor)to detect trailing vehicles and/or one more rear facing cameras thatprovide images to system 100 for analysis and/or traffic informationover wireless connection. Based on the number of vehicles travelingbehind vehicle 200, altruistic behavior module 804 may then evaluate apotential impact of whether or not vehicle 200 permits the targetvehicle to cut in. For example, if a long line of vehicles (e.g., 5, 10,or more vehicles) are determined to be traveling behind vehicle 200, ifvehicle 200 does not allow the target vehicle to cut in, it ispotentially unlikely that the target vehicle will have an opportunity tocut in until the trailing vehicles have passed, which may take asignificant amount of time. However, if a small number of vehicles(e.g., 1 or 2) are traveling behind vehicle 200, if vehicle 200 does notallow the target vehicle to cut in, the target vehicle will have anopportunity to cut in after a short period of time (e.g., after thesmall number of vehicles has passed).

In some embodiments, the altruistic behavior parameter may be derivedfrom examples using machine learning techniques such as neural networks.For example, a neural network could be trained to determine altruisticbehavior parameters based on scenarios, as described above. In someembodiments, the altruistic behavior parameters may be determined basedon previously detected driver behavior. For example, altruistic behaviorby drivers can be positively weighted, causing the neural networks topreference altruistic behavior. Inconsistency, as described above, canbe added through random or cyclical variation as described above.

Altruistic behavior module 806 is further described below in connectionwith FIGS. 10 and 12.

FIG. 9A is an illustration of an example situation in which vehicle 200may detect and respond to a cut in. As shown, vehicle 200 may betraveling on a roadway 900 along with a target vehicle 902. The targetvehicle 902 may be traveling in a first lane 904, while vehicle 200 maybe traveling in a second lane 906. While the first lane 904 is shown tobe the left lane and the second lane 906 is shown to be the right lane,it will be understood that the first and second lanes 904 and 906 may beany adjacent lanes on the roadway 900. Further, while only two lanes904, 906 are shown on the roadway 900, it will be understood that morelanes are possible as well. And while the term “lanes” is used forconvenience, in some situations (e.g., where lanes may not be clearlymarked with lane markers), the term “lane” may be understood to refer tomore generally to the pathways along with vehicle 200 and the targetvehicle 902 are traveling. For example, in some embodiments, referencesto a “lane” may refer to a path aligned with a travel direction or pathof vehicle 200.

Vehicle 200 may be configured to receive images of the environmentsurrounding the roadway 900 from, for example, one or more image capturedevices associated with vehicle 200, such as image capture devices 122,124, and/or 126. Vehicle 200 may receive other sensory information aswell, such as GPS data, map data, radar data, or lidar data. Based onthe images and/or the other sensory information, vehicle 200 may detectthe target vehicle 902. For example, vehicle 200 may identify arepresentation of the target vehicle 902 in the plurality of images.

Vehicle 200 may detect the target vehicle 902 in other manners as well,including any of the manners described above in connection with cut indetection module 802.

Further based on the images and/or the other sensory information,vehicle 200 may identify at least one indicator that the target vehicle902 will change from the first lane 904 to the second lane 906. Forexample, vehicle 200 may detect based on monocular and/or stereo imageanalysis of the images (or based on other sensory data, such as radar orlidar data) a position and/or speed of the target vehicle 902, asdescribed in connection with FIG. 5B, and/or a location of one or moreroad markings on the roadway 900, as described in connection with FIG.5C. As another example, vehicle 200 may detect that the target vehicle902 has begun to change lanes, as described above in connection withFIG. 5F. As still another example, vehicle 200 may detect based on mapdata that the first lane 904 is ending. Vehicle 200 may identify the atleast one indicator in other manners as well, including any of themanners described above in connection with cut in detection module 802.

As described above, once the indicator is detected, vehicle 200 maydetermine whether to undertake a navigational response. In order tominimize both unnecessary braking and sudden braking, vehicle 200 may,in making this determination, consider the presence or absence ofpredetermined cut in sensitivity change factors that affect thelikelihood the target vehicle 902 will cut in to the second lane 906.Where no predetermined cut in sensitivity change factor is detected, anavigational response may be caused in vehicle 200 based on theidentification of the indicator and based on a first cut in sensitivityparameter. On the other hand, where a predetermined sensitivity factoris detected, a navigational response may be caused in vehicle 200 basedon the identification of the indicator and based on a second cut insensitivity parameter. The second cut in sensitivity parameter may bedifferent than (e.g., more sensitive than) the first cut in sensitivityparameter.

Additional example situations involving predetermined cut in sensitivitychange factors are illustrated in FIGS. 9B-9E.

FIG. 9B illustrates an example predetermined cut in sensitivity changefactor that takes the form of an obstruction in the first lane 904. Asshown, the target vehicle 902 is traveling in the first lane 904, andvehicle 200 is traveling in the second lane 906. The obstruction mayserve as a predetermined cut in sensitivity change factor where theobstruction causes the target vehicle 902 to be more likely to attempt acut in than if an obstruction were not present.

As shown, the obstruction may be detected through detection of anothervehicle 908 traveling in the first lane 904 ahead of the target vehicle902. While the obstruction is illustrated to be the other vehicle 908,in some embodiments the obstruction may take the form of a stoppedvehicle, an accident, a hazard, a pedestrian, etc. In an embodiment inwhich the obstruction is another vehicle 908, vehicle 200 may detect theobstruction by detecting that the other vehicle 908 is traveling moreslowly than the target vehicle 902. This is because when the othervehicle 908 is traveling more slowly than the target vehicle 902, theslower speed of the other vehicle 908 makes it more likely that thetarget vehicle 902 will attempt a cut in into second lane 906. Such anobstruction may constitute a predetermined cut in sensitivity changefactor.

Vehicle 200 may detect the obstruction in any of the manners describedabove for detecting a sensitivity change factor in connection with FIG.8. For example, vehicle 200 may detect based on monocular and/or stereoimage analysis of the images a position and/or speed of the othervehicle 908 in addition to that of the target vehicle 902. The positionand/or speed of the other vehicle 908 may be detected based on othersensory information as well. As another example, vehicle 200 may detectthe obstruction (e.g., the other vehicle 908) based on GPS data, mapdata, and/or traffic data from, for instance, a traffic application suchas Waze. As yet another example, vehicle 200 may detect the obstructionvia analysis of radar or lidar data.

When the predetermined cut in sensitivity change factor is detected(that is, when vehicle 200 detects that there is an obstruction in thefirst lane 904), vehicle 200 may cause a navigational response based onthe identification of the indicator and based on a second cut insensitivity parameter. The navigational response may include, forexample, an acceleration of vehicle 200, a deceleration of vehicle 200,or (if possible) a lane change by vehicle 200. The second cut insensitivity parameter may be more sensitive than a first cut insensitivity parameter where no predetermined cut in sensitivity changefactor is detected, as the presence of the predetermined cut insensitivity change factor makes it more likely that the target vehicle902 will attempt a cut in into second lane 906.

In some embodiments, the second cut in sensitivity parameter may dependon the presence and behavior of vehicles surrounding vehicle 200. Forinstance, the second cut in sensitivity parameter may take into accounta lateral speed and lateral position of target vehicle 902, and asensitivity of the second cut in sensitivity parameter may be correlatedwith a threshold for each of the lateral speed and the lateral positionof target vehicle 902. A high lateral speed threshold and/or a lowlateral position threshold, for example, may result in a lowersensitivity, delaying a navigational response compared to a low lateralspeed threshold and/or a high lateral position threshold, which mayresult in a higher sensitivity and a quicker navigational response.

A higher sensitivity (that is, a more sensitive cut in sensitivityparameter) may be desirable in certain situations. For example, wheretarget vehicle 902 is moving more quickly than vehicle 200 and the othervehicle 908 is moving significantly more slowly than target vehicle 902in first lane 904, it is apparent that target vehicle 902 may have tomodify its behavior, either by slowing down and staying in first lane904, slowing down and changing into second lane 906 behind vehicle 200,changing into another lane to the left of first lane 904 (if such a laneexists), or cutting into second lane 906. If target vehicle 902 wouldhave to sharply decelerate to avoid colliding with the other vehicle908, a cut in to second lane 906 is more likely. Similarly, if targetvehicle 902 would have to sharply decelerate to change into second lane906 behind vehicle 200, a cut in to second lane 906 is more likely.

The sensitivity of the second cut in sensitivity parameter may furtherdepend on the presence and behavior of other vehicles surroundingvehicle 200. For example, where there is no other vehicle 908 in firstlane 904 and a distance between to vehicle 200 and the closest vehicleahead of vehicle 200 in second lane 906 is short, the sensitivity may belower. As another example, if the other vehicle 908 in first lane 904 ismoving at about the same speed as and/or more quickly than targetvehicle 902 and a distance between vehicle 200 and the closest vehicleahead of vehicle 200 in second lane 906 is short, the sensitivity may belower. As still another example, if target vehicle 902 is in a passinglane (e.g., if first lane 904 is to the left of second lane 906 in acountry where vehicles drive on the right side of roadway 900), theother vehicle 908 in first lane 904 is moving at about the same speed asand/or more quickly than target vehicle 902, and a distance betweenvehicle 200 and the closest vehicle ahead of vehicle 200 in second lane906 is large, the sensitivity may be slightly higher (e.g., a low tomoderate sensitivity). If, in the same situation, target vehicle 902 isnot in a passing lane (e.g., if the first lane 904 is to the left of thesecond lane 906 in a country where vehicles drive on the right side ofthe roadway 900), then the sensitivity may be lower.

The sensitivity of the second cut in sensitivity parameter may furthertake into account any acceleration of deceleration by target vehicle902. For example, if target vehicle 902 is moving more quickly thanother vehicle 908 in first lane 904 and target vehicle 902 accelerates,then the sensitivity may be increased. As another example, if targetvehicle 902 is moving more quickly than the other vehicle 908 in firstlane 904, the required deceleration of target vehicle 902 to avoid acollision with the other vehicle 908 is greater than, for instance, 0.1g, and target vehicle 902 is not decelerating, the sensitivity may beelevated, but not at a highest level. As still another example, iftarget vehicle 902 is moving more quickly than the other vehicle 908 infirst lane 904, the required deceleration of the target vehicle 902 toavoid a collision with the other vehicle 908 is greater than, forinstance, 0.5 g, and target vehicle 902 is not decelerating, thesensitivity may be at its highest. But if, in the same situation, theclosest vehicle ahead of vehicle 200 in second lane 906 is moving moreslowly than target vehicle 902 and the required deceleration of targetvehicle 902 to avoid a collision with the closest vehicle ahead ofvehicle 200 in second lane 906 is greater than that required to avoidhitting the other vehicle 908, the sensitivity may be high, but not thehighest. In the same situation, though, if a lane on the other side ofsecond lane 906 is free enough to permit a more gradual deceleration bytarget vehicle 902, the sensitivity may be at its highest, as targetvehicle 902 will likely attempt to cut in to second lane 906 to reachthe lane on the other side of second lane 906. It should be noted thatthe sensitivity levels described above and throughout the disclosure mayexist on a spectrum including any number of sensitivity levels arranged,for example, at any desired relative arrangement along the spectrum.

As another example, if target vehicle 902 is moving more quickly thanthe other vehicle 908 in first lane 904, the required deceleration oftarget vehicle 902 to avoid a collision with the other vehicle 908 isgreater than, for instance, 0.2 g, and target vehicle 902 decelerates tothe speed of vehicle 200 and/or the closest vehicle ahead of vehicle 200in second lane 906, sensitivity may be highest. As yet another example,if target vehicle 902 is moving more quickly than the other vehicle 908in first lane 904 and target vehicle 902 decelerates to below the speedof vehicle 200 and/or the closest vehicle ahead of vehicle 200 in secondlane 906, sensitivity may be low. As another example, if target vehicle902 is moving more quickly than the other vehicle 908 in first lane 904,the required deceleration of target vehicle 902 to avoid a collisionwith the other vehicle 908 is less than, for instance, 0.2 g, and targetvehicle 902 is decelerating, the sensitivity may be low.

In some embodiments, vehicle 200 may take into account other behavior oftarget vehicle 902 in determining the second cut in sensitivityparameter. For example, if target vehicle 902 is in a passing lane(e.g., if first lane 904 is to the left of second lane 906 in a countrywhere vehicles drive on the right side of roadway 900), the othervehicle 908 in first lane 904 is traveling more slowly than targetvehicle 902, and target vehicle 902 flashes its headlights at the othervehicle 908, a lower sensitivity may be used, as target vehicle 902 hasindicated that it intends to remain in first lane 904. As anotherexample, if target vehicle 902 activates its turn signal, indicatingthat it intends to attempt a cut in into second lane 906, sensitivitymay be high, but not the highest, as target vehicle 902 has indicatedthat it intends to cut in to second lane 906 but also that it is drivingor being driven cautiously. Following activation of the turn signal, forexample, a cut in may be detected only where target vehicle 902 exhibitssignificant lateral motion (e.g., 0.5 m towards the lane marker).

The navigational response undertaken by vehicle 200 may likewise dependon the presence and behavior of vehicles surrounding vehicle 200. Forexample, while in some cases vehicle 200 may decelerate to permit targetvehicle 902 to cut in to second lane 906, in other situations vehicle200 may accelerate to permit target vehicle 902 to change into secondlane 906 behind vehicle 200. Vehicle 200 may accelerate when, forinstance, vehicle 200 is traveling under the permitted speed, vehicle200 and target vehicle 902 are abreast and moving at approximately thesame speed, there is no vehicle ahead of vehicle 200 in second lane 906or the closest vehicle ahead of vehicle 200 in second lane 906 is at asafe distance, there is no other vehicle 908 ahead of target vehicle 902(or the other vehicle 908 is not moving more slowly than target vehicle902), or there is no other free lane into which target vehicle 902 couldchange. In some cases, vehicle 200 may accelerate quickly if required(e.g., if the required deceleration of target vehicle 902 to change intosecond lane 906 behind vehicle 200 is greater than, for instance, 0.5g).

FIG. 9C illustrates an example predetermined cut in sensitivity changefactor that takes the form of a geographic area. As shown, the targetvehicle 902 is traveling in the first lane 904, and vehicle 200 istraveling in the second lane 906. The geographic area may serve as apredetermined cut in sensitivity change factor where the geographic area(e.g., traffic rules and/or driving customs in the geographic area)causes the target vehicle 902 to be more likely to attempt a cut in thanthe target vehicle 902 would be were it not located in the geographicarea. In some embodiments, the geographical area may include a countryor other region with particular legal rules and/or driving customs thatgovern driving.

As shown, the geographic area may be detected through detection of roadsign 910 (e.g., using monocular and/or stereo image analysis of the roadsign 910) from which the geographic area may be ascertained.Alternatively or additionally, in some embodiments the geographic areamay be ascertained in other manners, such as through detection of one ormore geographic indicators or landmarks, using GPS or map data (or otherlocation determination system associated with vehicle 200), etc. Bydetecting the geographic area, vehicle 200 may determine whether trafficrules and/or driving customs in the geographic area will cause thetarget vehicle 902 to be more likely to attempt a cut in than the targetvehicle 902 would be were it not located in the geographic area. If so,the geographic area may constitute a predetermined cut in sensitivitychange factor.

When the predetermined cut in sensitivity change factor is detected(that is, when vehicle 200 detects that the target vehicle 902 istraveling in the geographic area), vehicle 200 may cause a navigationalresponse based on the identification of the indicator and based on asecond cut in sensitivity parameter. The navigational response mayinclude, for example, an acceleration of vehicle 200, a deceleration ofvehicle 200, or (if possible) a lane change by vehicle 200. The secondcut in sensitivity parameter may be more sensitive than a first cut insensitivity parameter where no predetermined cut in sensitivity changefactor is detected, as the presence of the cut in sensitivity changefactor makes it more likely that the target vehicle 902 will attempt acut in into second lane 906.

FIG. 9D illustrates an example predetermined cut in sensitivity changefactor that takes the form of an end of lane condition. As shown, thetarget vehicle 902 is traveling in the first lane 904, and vehicle 200is traveling in the second lane 906. The end of lane may serve as apredetermined cut in sensitivity change factor where the end of lanecondition causes the target vehicle 902 to be more likely to attempt acut in than the target vehicle 902 would be were the first lane 904 notending.

As shown, the end of lane condition may be detected through detection ofroad sign 912 (e.g., using monocular and/or stereo image analysis of theroad sign 912) and/or detection of road markings 914 (e.g., usingmonocular and/or stereo image analysis of the road sign 912) from whichthe end of lane condition may be ascertained. Alternatively oradditionally, in some embodiments the end of lane condition may beascertained in other manners, such as through using GPS or map data (orother location determination system associated with vehicle 200), etc.By detecting the end of lane condition, vehicle 200 may determine thatthe target vehicle 902 is more likely to attempt a cut in than thetarget vehicle 902 would be were the first lane 904 not ending.Accordingly, the end of lane condition may constitute a predeterminedcut in sensitivity change factor.

When the predetermined cut in sensitivity change factor is detected(that is, when vehicle 200 detects that the first lane 904 is ending),vehicle 200 may cause a navigational response based on theidentification of the indicator and based on a second cut in sensitivityparameter. The navigational response may include, for example, anacceleration of vehicle 200, a deceleration of vehicle 200, or (ifpossible) a lane change by vehicle 200. The second cut in sensitivityparameter may be more sensitive than a first cut in sensitivityparameter where no predetermined cut in sensitivity change factor isdetected, as the presence of the cut in sensitivity change factor makesit more likely that the target vehicle 902 will attempt a cut in intosecond lane 906.

FIG. 9E illustrates an example predetermined cut in sensitivity changefactor that takes the form of a roadway split condition. As shown, thetarget vehicle 902 is traveling in the first lane 904, and vehicle 200is traveling in the second lane 906. The roadway 900 may be splitting.The roadway split condition may serve as a predetermined cut insensitivity change factor where the roadway split condition causes thetarget vehicle 902 to be more likely to attempt a cut in than the targetvehicle 902 would be were the first lane 904 not ending.

As shown, the roadway split condition may be detected through detectionof road sign 916 (e.g., using monocular and/or stereo image analysis ofthe road sign 912) and/or detection of road markings 918a, 918b (e.g.,using monocular and/or stereo image analysis of the road sign 912) fromwhich the roadway split condition may be ascertained. Alternatively oradditionally, in some embodiments the roadway split condition may beascertained in other manners, such as through using GPS or map data (orother location determination system associated with vehicle 200), etc.By detecting roadway split condition, vehicle 200 may determine that thetarget vehicle 902 is more likely to attempt a cut in than the targetvehicle 902 would be were the roadway 900 not splitting. Accordingly,roadway split condition may constitute a predetermined cut insensitivity change factor.

When the predetermined cut in sensitivity change factor is detected(that is, when vehicle 200 detects that the roadway 900 is splitting),vehicle 200 may cause a navigational response based on theidentification of the indicator and based on a second cut in sensitivityparameter. The navigational response may include, for example, anacceleration of vehicle 200, a deceleration of vehicle 200, or (ifpossible) a lane change by vehicle 200. The second cut in sensitivityparameter may be more sensitive than a first cut in sensitivityparameter where no predetermined cut in sensitivity change factor isdetected, as the presence of the predetermined cut in sensitivity changefactor makes it more likely that the target vehicle 902 will attempt acut in into second lane 906.

While certain predetermined cut in sensitivity change factors have beenillustrated, it will be understood that other predetermined cut insensitivity change factors are possible as well, including anyenvironmental factor that may cause and/or contribute to conditions thatcause a cut in by the target vehicle 902 to be more or less likely thanit would otherwise be.

FIG. 10 is an illustration of an example situation in which vehicle 200may engage in altruistic behavior, consistent with the disclosedembodiments. As shown, vehicle 200 may be traveling on a roadway 1000along with a target vehicle 1002. Vehicle 200 may be traveling in afirst lane 1004, while the target vehicle 1002 may be traveling in asecond lane 1006. While the first lane 1004 is shown to be the left laneand the second lane 1006 is shown to be the right lane, it will beunderstood that the first and second lanes 1004 and 1006 may be anyadjacent lanes on the roadway 1000. Further, while only two lanes 1004,1006 are shown on the roadway 1000, it will be understood that morelanes are possible as well. And while the term “lanes” is used forconvenience, in some situations (e.g., where lanes may not be clearlymarked with lane markers), the term “lane” may be understood to refer tomore generally to the pathways along with vehicle 200 and the targetvehicle 1002 are traveling.

Vehicle 200 may be configured to receive images of the environmentsurrounding the roadway 1000 from, for example, one or more imagecapture devices associated with vehicle 200, such as image capturedevices 122, 124, and/or 126. Vehicle 200 may receive other sensoryinformation as well, such as GPS or map data. Further, in someembodiments, vehicle 200 may receive other sensory information, such asinformation acquired via a radar device or lidar device included insystem 100. Based on the images and/or the other sensory information,vehicle 200 may detect the target vehicle 1002. For example, vehicle 200may identify a representation of the target vehicle 1002 in theplurality of images. Vehicle 200 may detect the target vehicle 1002 inother manners as well, including any of the manners described above inconnection with cut in detection module 802.

Further based on the images and/or the other sensory information,vehicle 200 may determine one or more situational characteristicsassociated with the target vehicle 1002. A situational characteristicmay be, for example, any characteristic that indicates the targetvehicle 1002 would benefit from changing lanes into the lane in whichvehicle 200 is traveling. For example, as shown, a situationalcharacteristic may indicate that the target vehicle 1002 is travelingbehind another vehicle 1008 that is traveling more slowly than thetarget vehicle 1002. In general, the situational characteristic mayindicate that, while a cut in or other navigational response by thetarget vehicle 1002 may not be necessary, a cut in would benefit thetarget vehicle 1002.

In some embodiments, vehicle 200 may detect based on monocular and/orstereo image analysis of the images a position and/or speed of thetarget vehicle 1002, as described in connection with FIG. 5B, and/or alocation of one or more road markings on the roadway 900, as describedin connection with FIG. 5C. As another example, vehicle 200 may detectthat the target vehicle 1002 has begun to change lanes, as describedabove in connection with FIG. 5F. As yet another example, vehicle 200may detect that target vehicle 1002 has begun to change lanes based onanalysis of other sensory information, such as information acquired viaa radar device or lidar device included in system 100. Vehicle 200 mayidentify the one or more situational characteristics in other manners aswell, including any of the manners described above in connection withaltruistic behavior module 806.

When the one or more situational characteristics are detected, vehicle200 may determine a current value associated with an altruistic behaviorparameter in order to determine whether vehicle 200 will allow thetarget vehicle 1002 to cut in to the first lane 1004. The altruisticbehavior parameter may take any of the forms described above inconnection with the altruistic behavior module 806. For example, thealtruistic behavior parameter may be set based on input from an operatorof vehicle 200, based on at least one informational element determinedby parsing calendar entries for an operator or a passenger of vehicle200, based on an output of a randomizer function, and/or based on adetermined number of encounters with target vehicles for which the oneor more situational characteristics indicate that the target vehiclewould benefit from a course change into the first lane 1004. Thealtruistic behavior parameter may be fixed or may be updated such that anavigational change in vehicle 200 is caused in at least a predeterminedpercentage of encounters with target vehicles for which the one or moresituational characteristics indicate that the target vehicle wouldbenefit from a course change into a path of vehicle 200. For example,the predetermined percentage may be, e.g., at least 10%, at least 20%,at least 30%, etc.

Vehicle 200 may determine based on the one or more situationalcharacteristics associated with the target vehicle 1002 that a change inthe navigation state of vehicle 200 may not be necessary. That is, theone or more situational characteristics may indicate that, while itwould benefit the target vehicle 1002 to cut into the first lane 1004,such a cut in may not be necessary (e.g., by traffic rules or forsafety). Nevertheless, the target vehicle 1002 may, in some instances,cause at least one navigational change in vehicle 200 to permit thetarget vehicle 1002 to cut into the first lane 1004, based on thealtruistic behavior parameter and the one or more situationalcharacteristics.

For example, where the altruistic behavior parameter may be set based oninput from an operator of vehicle 200, the operator may provide inputindicating that vehicle 200 should allow the target vehicle 1002 to cutin. Based on the altruistic behavior parameter and the one or moresituational characteristics, the target vehicle 1002 may alter its speedto allow the target vehicle 1002 to cut into the first lane 1004. Asanother example, where the altruistic behavior parameter may be setbased on an output of a randomizer function, the randomizer function mayprovide an output indicating that vehicle 200 may not allow the targetvehicle 1002 to cut into the first lane 1004. For example, therandomizer function may output a binary output (e.g., “NO” or “0”) ormay output a value output that doesn't satisfy a certain threshold(e.g., may output a “2” where the threshold is “>=5”). Based on thealtruistic behavior parameter and the one or more situationalcharacteristics, the target vehicle 1002 may maintain its speed toprevent the target vehicle 1002 from cutting into the first lane 1004.As still another example, where the altruistic behavior parameter is setbased on at least one informational element determined by parsingcalendar entries for the operator of vehicle 200, the operator'scalendar entries may indicate that the operator wishes to arrive at adestination by a desired time. The altruistic behavior parameter mayindicate that the target vehicle 1002 should be let in so long as theoperator will still arrive at the destination by the desired time. Basedon the altruistic behavior parameter and the one or more situationalcharacteristics, the target vehicle 1002 effect a navigational change topermit the target vehicle 1002 to cut into the first lane 1004 if doingso will not prevent the operator from arriving at the destination at thedesired time, but will not permit the target vehicle 1002 to cut intothe first lane 1004 if doing so will prevent the operator from arrivingat the destination at the desired time.

In some cases, more than one target vehicle may benefit from cuttinginto the first lane 1004. For example, as shown, in addition to targetvehicle 1002, an additional target vehicle 1010 may be traveling in thesecond lane 1006 behind the other vehicle 1008. The one or moresituational characteristics may indicate that, like the target vehicle1002, while a cut in or other navigational response by the additionaltarget vehicle 1010 may not be necessary, a cut in would benefit theadditional target vehicle 1010. For example, as shown, the additionaltarget vehicle 1010 may also traveling behind the other vehicle 1008,and the other vehicle 1008 may be traveling more slowly than theadditional target vehicle 1010. In these cases, the altruistic behaviorparameter may cause vehicle 200 to treat each of the target vehicle 1002and the additional target vehicle 1010 the same (i.e., allowing both orneither to cut into the first lane 1004), or the altruistic behaviorparameter may cause vehicle 200 to treat the target vehicle 1002 and theadditional target vehicle 1010 inconsistently.

For example, the altruistic behavior parameter may be set based on thenumber of encounters with the target vehicles, such as vehicles 1002,1010 for which the situational characteristics indicate that the targetvehicle 1002 or 1010 would benefit from a course change into the firstlane 1004. For instance, with two target vehicles 1002, 1010, thealtruistic behavior parameter may cause vehicle 200 to let only thetarget vehicle 1002 cut into the first lane 1004, but not the additionaltarget vehicle 1010 (whereas the altruistic behavior parameter may haveallowed both of the target vehicle 1002 and the additional targetvehicle 1010 to cut in if encountered alone).

As another example, in these cases the altruistic behavior parameter maybe updated such that a navigational change in vehicle 200 is caused inat least a predetermined percentage of encounters with the targetvehicles 1002, 1010 for which the situational characteristics indicatethat the target vehicle 1002 or 1010 would benefit from a course changeinto the first lane 1004. For instance, the altruistic behaviorparameter may specify that vehicle 200 should effect a navigationalchange in at least 10% of encounters. Based on the altruistic behaviorparameter, vehicle 200 may allow in one or both of the target vehicle1002 and the additional target vehicle 1010 so long as an overallpercentage of encounters, continuously determined, cause vehicle 200 toeffect a navigational change in at least 10% of encounters.

Furthermore, although consideration of an altruistic behavior parameterhas been discussed above in connection with the example shown in FIG.10, an altruistic behavior parameter may be taken into account under anyof the examples discussed above in connection with FIGS. 9A-9E.

FIG. 11 is a flowchart showing an exemplary process 1100 for vehicle cutin detection and response, consistent with disclosed embodiments. Insome embodiments, processing unit 110 of system 100 may execute one ormore of modules 402-408 and 802-806. In other embodiments, instructionsstored in one or more of modules 402-408 and 802-806 may be executedremotely from system 100 (e.g., vi a server accessible over a networkvia wireless transceiver 172). In still yet other embodiments,instructions associated with one or more of modules 402-408 and 802-806may be executed by processing unit 110 and a remote server.

As shown, process 1100 includes, at step 1102, receiving images. Forexample, vehicle 200 may receive, via a data interface, a plurality ofimages from at least one image capture device (e.g., image capturedevices 122, 124, 126) associated with vehicle 200. As discussed above,in other embodiments, vehicle 200 may instead analyze other sensoryinformation, such as information acquired via a radar device or lidardevice included in system 100, as an alternative to or in addition toanalyzing images.

At step 1104, process 1100 includes identifying a target vehicle. Forexample, vehicle 200 may identify, in the plurality of images, arepresentation of a target vehicle traveling in a first lane differentfrom a second lane in which vehicle 200 is traveling. Identifying therepresentation of the target vehicle may involve, for example, monocularor stereo image analysis and/or other sensory information, as describedabove in connection with cut in detection module 802. In otherembodiments, as discussed above, identifying the representation of thetarget vehicle may involve analyzing other sensory information, such asinformation acquired via a radar device or lidar device included insystem 100, as an alternative to or in addition to analyzing images.

At step 1106, process 1100 includes identifying an indicator that thetarget vehicle will change lanes. For example, vehicle 200 may identify,based on analysis of the plurality of images, at least one indicatorthat the target vehicle will change from the first lane to the secondlane. Identifying the indicator may involve, for example, monocular orstereo image analysis and/or other sensory information (e.g., radar orlidar data), as described above in connection with cut in detectionmodule 802.

At step 1108, process 1100 includes determining whether a predeterminedcut in sensitivity change factor is present. For example, vehicle 200may determine whether at least one predetermined cut in sensitivitychange factor is present in an environment of vehicle 200. The at leastone predetermined cut in sensitivity change factor may, for example,take any of the forms described above in connection with FIGS. 9A-E. Thepredetermined cut in sensitivity change factor may include, for example,an end of lane condition, an obstruction in a path of the targetvehicle, a roadway split, or a geographic area. Detecting the at leastone predetermined cut in sensitivity change factor may involve, forexample, monocular or stereo image analysis and/or other sensoryinformation (e.g., rada or lidar data), as described above in connectionwith cut in detection module 802.

If no predetermined cut in sensitivity change factor is detected,process 1100 may continue at step 1110 with causing a first navigationalresponse based on the indicator and a first cut in sensitivityparameter. For example, vehicle 200 may cause the first navigationalresponse in the vehicle based on the identification of the at least oneindicator and based on the first cut in sensitivity parameter where nocut in sensitivity change factor is detected. The first cut insensitivity parameter may take any of the forms described above inconnection with cut in response module 804. Further, as discussed above,in some embodiments, step 1110 may instead include causing a firstnavigational response based on the indicator and a value associated witha first predetermined cut in sensitivity parameter, the firstnavigational response may take any of the forms described fornavigational responses in connection with FIG. 4.

If, on the other hand, at least one predetermined cut in sensitivitychange factor is detected, process 1100 may continue at step 1112 withcausing a second navigational response based on the indicator and asecond cut in sensitivity parameter. For example, vehicle 200 may causethe second navigational response in the vehicle based on theidentification of the at least one indicator and based on the second cutin sensitivity parameter where a cut in sensitivity change factor isdetected. The second cut in sensitivity parameter may take any of theforms described above in connection with cut in response module 804. Thesecond cut in sensitivity parameter may be different than (e.g., moresensitive than) the first cut in sensitivity parameter. Further, asdiscussed above, in some embodiments, step 1112 may instead includecausing a second navigational response based on the indicator and avalue associated with a second predetermined cut in sensitivityparameter, The second navigational response may take any of the formsdescribed for navigational responses in connection with FIG. 4.

FIG. 12 is a flowchart showing an exemplary process 1200 for navigatingwhile taking into account altruistic behavioral considerations,consistent with disclosed embodiments. In some embodiments, processingunit 110 of system 100 may execute one or more of modules 402-408 and802-806. In other embodiments, instructions stored in one or more ofmodules 402-408 and 802-806 may be executed remotely from system 100(e.g., vi a server accessible over a network via wireless transceiver172). In still yet other embodiments, instructions associated with oneor more of modules 402-408 and 802-806 may be executed by processingunit 110 and a remote server.

As shown, process 1200 includes at step 1202 receiving images. Forexample, vehicle 200 may receive, via a data interface, a plurality ofimages from at least one image capture device (e.g., image capturedevices 122, 124, 126) associated with vehicle 200. As discussed above,in other embodiments, vehicle 200 may instead analyze other sensoryinformation, such as information acquired via a radar device or lidardevice included in system 100, as an alternative to or in addition toanalyzing images.

Process 1200 includes at step 1204 identifying a target vehicle. Forexample, vehicle 200 may identify, based on an analysis of the pluralityof images, at least one target vehicle in an environment of vehicle 200.Identifying the target vehicle may involve, for example, monocular orstereo image analysis and/or other sensory information (e.g., radar orlidar data), as described above in connection with altruistic behaviormodule 806.

At step 1206, process 1200 includes determining one or more situationalcharacteristics associated with the target vehicle. For example, vehicle200 may determine, based on analysis of the plurality of images, one ormore situational characteristics associated with the target vehicle. Thesituational characteristic(s) may take any of the forms described abovein connection with the altruistic behavior module 806 and/or FIGS. 9A-Eand 10. For example, the situational characteristic(s) may indicate thatthe target vehicle would benefit from a course change into a path ofvehicle 200. As another example, the situational characteristic(s)mayindicate that the target vehicle is traveling in a lane adjacent to alane in which vehicle 200 is traveling and that the target vehicle isbehind a vehicle moving more slowly than the target vehicle and moreslowly than vehicle 200. Determining the situational characteristic(s)may involve, for example, monocular or stereo image analysis and/orother sensory information (e.g., rada or lidar data), as described abovein connection with altruistic behavior module 806.

At step 1208, process 1200 includes determining a current value of analtruistic behavior parameter. For example, vehicle 200 may determine acurrent value associated with the altruistic behavior parameter. Thealtruistic behavior parameter may take any of the forms described abovein connection with the altruistic behavior module 806 and/or FIG. 10,and determining the altruistic behavior parameter may be done in any ofthe manners described above in connection with the altruistic behaviormodule 806 and/or FIG. 10. The value of the altruistic behaviorparameter may be set based on input from an operator of vehicle 200,based on at least one informational element determined by parsingcalendar entries for an operator of vehicle 200, based on an output of arandomizer function, and/or based on a determined number of encounterswith target vehicles for which the one or more situationalcharacteristics indicate that the target vehicle would benefit from acourse change into a path of vehicle 200. Alternatively or additionally,the value of the altruistic behavior parameter may be updated such thata navigational change in vehicle 200 is caused in at least apredetermined percentage (e.g., 10%, 20%, 30%, etc.) of encounters withtarget vehicles for which the one or more situational characteristicsindicate that the target vehicle would benefit from a course change intoa path of vehicle 200.

At step 1210, process 1200 includes causing a navigational change basedon the one or more situational characteristics and the current value ofthe altruistic behavior parameter. For example, vehicle 200 maydetermine based on the one or more situational characteristicsassociated with the target vehicle that no change in a navigation stateof vehicle 200 may be necessary, but may nevertheless cause at least onenavigational change in vehicle 200 based on the current value associatedwith the altruistic behavior parameter and based on the one or moresituational characteristics associated with the target vehicle. Causingthe navigational change may be done in any of the manners describedabove in connection with the altruistic behavior module 806 and/or FIG.10.

As discussed above, in any process or process step described in thisdisclosure, in addition to performing image analysis of images capturedfrom one or more front and/or rear facing cameras, system 100 (includedin vehicle 200) may analyze other sensory information, such asinformation acquired via a radar device and/or a lidar device.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer readable media,such as secondary storage devices, for example, hard disks or CD ROM, orother forms of RAM or ROM, USB media, DVD, Blu-ray, 4K Ultra HD Blu-ray,or other optical drive media.

Computer programs based on the written description and disclosed methodsare within the skill of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of .Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with includedJava applets.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. Thelimitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application.The examples are to be construed as non-exclusive. Furthermore, thesteps of the disclosed methods may be modified in any manner, includingby reordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asillustrative only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

1.-20. (canceled)
 21. A vehicle navigation system comprising: a memoryincluding instructions; and circuitry that, when in operation, isconfigured by the instructions to: identify a target vehicle in anenvironment of a vehicle that includes the vehicle navigation system;receive image data of the target vehicle from at least one image capturedevice of the vehicle; identify, based on analysis of the image data,one or more situational characteristics of the target vehicle, thesituational characteristics of the target including an indication thatthe target vehicle has an activated blinker; and change a navigationalstate of the vehicle to allow an action of the target vehicle, thevehicle configured to cause the change in the navigational state basedon a determination that the one or more situational characteristics,including the activated blinker, indicate that the target vehicle wouldbenefit from the change in the navigational state.
 22. The vehiclenavigation system of claim 21, wherein the one or more situationalcharacteristics indicate that the target vehicle would benefit from acourse change into a path of the vehicle.
 23. The vehicle navigationsystem of claim 22, wherein the one or more situational characteristicsindicate that the target vehicle has moved laterally towards the path ofthe vehicle.
 24. The vehicle navigation system of claim 21, wherein theone or more situational characteristics indicate that the target vehicleis traveling in a lane adjacent to a lane in which the vehicle istraveling.
 25. The vehicle navigation system of claim 21, wherein theaction is a cut-in to a lane in which the vehicle is traveling.
 26. Thevehicle navigation system of claim 25, wherein the action is a cut-in tothe lane in which the vehicle is traveling occurs at an interchange. 27.The vehicle navigation system of claim 21, wherein, to change thenavigational state of the vehicle, the circuitry is configured to altera speed of the vehicle.
 28. The vehicle navigation system of claim 21,wherein, to identify the target vehicle in the environment of thevehicle, the circuitry is configured to use radar of the vehicle.
 29. Avehicle comprising: at least one image capture device; and a vehiclenavigation system configured to: identify a target vehicle in anenvironment of the vehicle; receive image data of the target vehiclefrom the at least one image capture device; identify, based on analysisof the image data, one or more situational characteristics of the targetvehicle, the situational characteristics of the target including anindication that the target vehicle has an activated blinker; and changea navigational state of the vehicle to allow an action of the targetvehicle, the vehicle configured to cause the change in the navigationalstate based on a determination that the one or more situationalcharacteristics, including the activated blinker, indicate that thetarget vehicle would benefit from the change in the navigational state.30. The vehicle of claim 29, wherein the one or more situationalcharacteristics indicate that the target vehicle would benefit from acourse change into a path of the vehicle.
 31. The vehicle of claim 30,wherein the one or more situational characteristics indicate that thetarget vehicle has moved laterally towards the path of the vehicle. 32.The vehicle navigation system of claim 29, wherein the one or moresituational characteristics indicate that the target vehicle istraveling in a lane adjacent to a lane in which the vehicle istraveling.
 33. The vehicle of claim 29, wherein the action is a cut-into a lane in which the vehicle is traveling.
 34. The vehicle of claim33, wherein the action is a cut-in to the lane in which the vehicle istraveling occurs at an interchange.
 35. The vehicle of claim 29,wherein, to change the navigational state of the vehicle, the vehiclenavigation system is configured to alter a speed of the vehicle.
 36. Thevehicle of claim 29, further comprising a radar; and wherein, toidentify the target vehicle in the environment of the vehicle, thevehicle navigation system is configured to use radar of the vehicle. 37.At least one non-transitory machine-readable medium includinginstructions that, when executed by circuitry of a vehicle component fora vehicle, cause the circuitry to perform operations comprising:identifying a target vehicle in an environment of the vehicle; receivingimage data of the target vehicle from at least one image capture deviceof the vehicle; identifying, based on analysis of the image data, one ormore situational characteristics of the target vehicle, the situationalcharacteristics of the target including an indication that the targetvehicle has an activated blinker; and changing a navigational state ofthe vehicle to allow an action of the target vehicle, the vehicleconfigured to cause the change in the navigational state based on adetermination that the one or more situational characteristics,including the activated blinker, indicate that the target vehicle wouldbenefit from the change in the navigational state.
 38. The at least onenon-transitory machine-readable medium of claim 37, wherein the one ormore situational characteristics indicate that the target vehicle wouldbenefit from a course change into a path of the vehicle.
 39. The atleast one non-transitory machine-readable medium of claim 38, whereinthe one or more situational characteristics indicate that the targetvehicle has moved laterally towards the path of the vehicle.
 40. The atleast one non-transitory machine-readable medium of claim 37, whereinthe one or more situational characteristics indicate that the targetvehicle is traveling in a lane adjacent to a lane in which the vehicleis traveling.
 41. The at least one non-transitory machine-readablemedium of claim 37, wherein the action is a cut-in to a lane in whichthe vehicle is traveling.
 42. The at least one non-transitorymachine-readable medium of claim 41, wherein the action is a cut-in tothe lane in which the vehicle is traveling occurs at an interchange. 43.The at least one non-transitory machine-readable medium of claim 37,wherein changing the navigational state of the vehicle includes alteringa speed of the vehicle.
 44. The at least one non-transitorymachine-readable medium of claim 37, wherein identifying the targetvehicle in the environment of the vehicle includes using radar of thevehicle.